==================== Test output for //tensorflow/python/distribute/failure_handling:gce_failure_handler_test (shard 7 of 8): 2023-05-20 05:20:01.592713: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. Running tests under Python 3.10.11: /usr/local/bin/python3 [ RUN ] GceFailureHandlingTest.test_basic_run_test_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 24960 I0520 05:20:08.071786 281473279947648 test_util.py:3796] Using local port 24960 INFO:tensorflow:Using local port 18949 I0520 05:20:08.072573 281473279947648 test_util.py:3796] Using local port 18949 INFO:tensorflow:Using local port 21439 I0520 05:20:08.072987 281473279947648 test_util.py:3796] Using local port 21439 INFO:tensorflow:Using local port 22138 I0520 05:20:08.073396 281473279947648 test_util.py:3796] Using local port 22138 2023-05-20 05:20:09.501980: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2023-05-20 05:20:11.262015: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. INFO:tensorflow:Cluster starting. I0520 05:20:14.451282 281473279947648 gce_failure_handler_test.py:317] Cluster starting. [worker-0]: I0520 05:20:14.520226 281473412068224 multi_process_runner.py:840] Subprocess with PID 460947 (worker, 0) is now being started. [worker-0]: I0520 05:20:14.520674 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24960", "localhost:18949", "localhost:21439", "localhost:22138"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-0]: 2023-05-20 05:20:14.789668: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24960 [worker-1]: I0520 05:20:14.819021 281473412068224 multi_process_runner.py:840] Subprocess with PID 461388 (worker, 1) is now being started. [worker-0]: 2023-05-20 05:20:14.826205: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 17580468524868123442 [worker-0]: 2023-05-20 05:20:14.827113: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-1]: I0520 05:20:14.819494 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24960", "localhost:18949", "localhost:21439", "localhost:22138"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0520 05:20:14.846695 281473412068224 multi_process_runner.py:840] Subprocess with PID 462654 (worker, 2) is now being started. [worker-2]: I0520 05:20:14.847162 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24960", "localhost:18949", "localhost:21439", "localhost:22138"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0520 05:20:14.844397 281473412068224 multi_process_runner.py:840] Subprocess with PID 462665 (worker, 3) is now being started. [worker-3]: I0520 05:20:14.844880 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24960", "localhost:18949", "localhost:21439", "localhost:22138"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-3]: 2023-05-20 05:20:14.904594: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:22138 [worker-2]: E0520 05:20:14.926058640 462654 server_chttp2.cc:40] {"created":"@1684560014.925921661","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1684560014.925916810","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1684560014.925886682","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684560014.925871956","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1684560014.925914945","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684560014.925908394","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-2]: 2023-05-20 05:20:14.926263: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:608] UNKNOWN: Could not start gRPC server [worker-2]: 2023-05-20 05:20:14.929870: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:699] Could not start gRPC server [worker-0]: 2023-05-20 05:20:14.946230: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 3610190449304886341 [worker-3]: 2023-05-20 05:20:14.947511: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-2]: Process _Process-4: [worker-2]: Traceback (most recent call last): [worker-2]: File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-2]: self.run() [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-2]: return self._actual_run() [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-2]: app.run(lambda _: self._run_impl()) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-2]: _run_main(main, args) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-2]: sys.exit(main(argv)) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-2]: app.run(lambda _: self._run_impl()) [worker-2]: File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-2]: self._target(*self._args, **self._kwargs) [worker-1]: 2023-05-20 05:20:14.967127: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:18949 [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-2]: six.reraise(*info.exc_info) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-2]: raise value [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-2]: return_value = fn(*args, **kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 134, in worker_fn [worker-2]: strategy = collective_all_reduce_strategy.CollectiveAllReduceStrategy() [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 186, in __init__ [worker-2]: CollectiveAllReduceExtended( [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 339, in __init__ [worker-2]: self._initialize_strategy(self._cluster_resolver, devices=devices) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 358, in _initialize_strategy [worker-2]: self._initialize_multi_worker(cluster_resolver) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 530, in _initialize_multi_worker [worker-2]: context.context().ensure_initialized() [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized [worker-2]: pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) [worker-2]: tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server [worker-0]: 2023-05-20 05:20:14.984180: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 8618337472001919601 [worker-1]: 2023-05-20 05:20:14.984448: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. INFO:tensorflow:restarting workers I0520 05:20:44.927496 281473279947648 gce_failure_handler_test.py:323] restarting workers INFO:tensorflow:workers restarted I0520 05:20:45.101365 281473279947648 gce_failure_handler_test.py:327] workers restarted [worker-0]: I0520 05:20:45.367575 281473412068224 multi_process_runner.py:840] Subprocess with PID 517633 (worker, 0) is now being started. [worker-0]: I0520 05:20:45.367934 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24960", "localhost:18949", "localhost:21439", "localhost:22138"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-0]: E0520 05:20:45.417833605 517633 server_chttp2.cc:40] {"created":"@1684560045.417702007","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1684560045.417696546","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1684560045.417668843","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684560045.417652791","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1684560045.417694926","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684560045.417688495","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-0]: 2023-05-20 05:20:45.417958: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:608] UNKNOWN: Could not start gRPC server [worker-0]: 2023-05-20 05:20:45.426548: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:699] Could not start gRPC server [worker-1]: I0520 05:20:45.472089 281473412068224 multi_process_runner.py:840] Subprocess with PID 517696 (worker, 1) is now being started. [worker-2]: I0520 05:20:45.473873 281473412068224 multi_process_runner.py:840] Subprocess with PID 517708 (worker, 2) is now being started. [worker-2]: I0520 05:20:45.474205 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24960", "localhost:18949", "localhost:21439", "localhost:22138"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-1]: I0520 05:20:45.472463 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24960", "localhost:18949", "localhost:21439", "localhost:22138"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-3]: I0520 05:20:45.504193 281473412068224 multi_process_runner.py:840] Subprocess with PID 517889 (worker, 3) is now being started. [worker-0]: Process _Process-6: [worker-3]: I0520 05:20:45.504564 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24960", "localhost:18949", "localhost:21439", "localhost:22138"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-3]: E0520 05:20:45.527883331 517889 server_chttp2.cc:40] {"created":"@1684560045.527753402","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1684560045.527747896","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1684560045.527720114","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684560045.527704962","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1684560045.527746756","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684560045.527740011","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-3]: 2023-05-20 05:20:45.528010: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:608] UNKNOWN: Could not start gRPC server [worker-0]: Traceback (most recent call last): [worker-0]: File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: self.run() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-0]: return self._actual_run() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-0]: app.run(lambda _: self._run_impl()) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-0]: _run_main(main, args) [worker-1]: E0520 05:20:45.568182679 517696 server_chttp2.cc:40] {"created":"@1684560045.568052461","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1684560045.568047180","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1684560045.568019467","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684560045.568003706","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1684560045.568045880","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684560045.568038904","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-1]: 2023-05-20 05:20:45.568316: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:608] UNKNOWN: Could not start gRPC server [worker-1]: 2023-05-20 05:20:45.569167: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:699] Could not start gRPC server [worker-0]: sys.exit(main(argv)) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-0]: app.run(lambda _: self._run_impl()) [worker-0]: File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-0]: six.reraise(*info.exc_info) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-0]: raise value [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-0]: return_value = fn(*args, **kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 134, in worker_fn [worker-0]: strategy = collective_all_reduce_strategy.CollectiveAllReduceStrategy() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 186, in __init__ [worker-0]: CollectiveAllReduceExtended( [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 339, in __init__ [worker-0]: self._initialize_strategy(self._cluster_resolver, devices=devices) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 358, in _initialize_strategy [worker-0]: self._initialize_multi_worker(cluster_resolver) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 530, in _initialize_multi_worker [worker-0]: context.context().ensure_initialized() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized [worker-0]: pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) [worker-0]: tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server [worker-2]: 2023-05-20 05:20:45.737401: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:21439 [worker-0]: 2023-05-20 05:20:45.757036: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 1127801160307618450 [worker-1]: Process _Process-7: [worker-2]: 2023-05-20 05:20:45.806150: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-1]: Traceback (most recent call last): [worker-1]: File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: self.run() [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0520 05:20:45.817203 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0520 05:20:45.808585 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: I0520 05:20:45.818231 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-1]: return self._actual_run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-1]: app.run(lambda _: self._run_impl()) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-1]: _run_main(main, args) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-1]: sys.exit(main(argv)) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-1]: app.run(lambda _: self._run_impl()) [worker-1]: File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-1]: six.reraise(*info.exc_info) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-1]: raise value [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-1]: return_value = fn(*args, **kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 134, in worker_fn [worker-1]: strategy = collective_all_reduce_strategy.CollectiveAllReduceStrategy() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 186, in __init__ [worker-1]: CollectiveAllReduceExtended( [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 339, in __init__ [worker-1]: self._initialize_strategy(self._cluster_resolver, devices=devices) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 358, in _initialize_strategy [worker-1]: self._initialize_multi_worker(cluster_resolver) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 530, in _initialize_multi_worker [worker-1]: context.context().ensure_initialized() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized [worker-1]: pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) [worker-1]: tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0520 05:20:45.826884 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0520 05:20:45.880141 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0520 05:20:45.880624 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24960', 'localhost:18949', 'localhost:21439', 'localhost:22138']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0520 05:20:45.880817 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24960', 'localhost:18949', 'localhost:21439', 'localhost:22138']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0520 05:20:45.967669 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0520 05:20:45.968984 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24960', 'localhost:18949', 'localhost:21439', 'localhost:22138']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0520 05:20:45.969636 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24960', 'localhost:18949', 'localhost:21439', 'localhost:22138']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0520 05:20:46.042620 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0520 05:20:46.043775 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24960', 'localhost:18949', 'localhost:21439', 'localhost:22138']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0520 05:20:46.044431 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24960', 'localhost:18949', 'localhost:21439', 'localhost:22138']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0520 05:20:46.095267 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0520 05:20:46.095876 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24960', 'localhost:18949', 'localhost:21439', 'localhost:22138']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0520 05:20:46.096083 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24960', 'localhost:18949', 'localhost:21439', 'localhost:22138']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0520 05:20:46.307710 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0520 05:20:46.312344 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0520 05:20:46.326787 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0520 05:20:46.327770 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0520 05:20:46.330134 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0520 05:20:46.339865 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0520 05:20:46.340433 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0520 05:20:46.340991 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0520 05:20:46.341175 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0520 05:20:46.346799 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-0]: INFO:tensorflow:['workertemp_2', 'workertemp_1', 'workertemp_3'] [worker-0]: I0520 05:20:46.357790 281473412068224 gce_failure_handler_test.py:203] ['workertemp_2', 'workertemp_1', 'workertemp_3'] [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0520 05:20:46.356321 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-1]: I0520 05:20:46.376163 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: I0520 05:20:46.376589 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: W0520 05:20:46.376552 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-3]: Instructions for updating: [worker-2]: I0520 05:20:46.360195 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0520 05:20:46.360575 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0520 05:20:46.377090 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0520 05:20:46.376711 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0520 05:20:46.377269 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: I0520 05:20:46.360732 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: INFO:tensorflow:['workertemp_2', 'workertemp_1', 'workertemp_3'] [worker-2]: I0520 05:20:46.391801 281473412068224 gce_failure_handler_test.py:203] ['workertemp_2', 'workertemp_1', 'workertemp_3'] [worker-1]: INFO:tensorflow:['workertemp_2', 'workertemp_1', 'workertemp_3'] [worker-1]: I0520 05:20:46.458187 281473412068224 gce_failure_handler_test.py:203] ['workertemp_2', 'workertemp_1', 'workertemp_3'] [worker-3]: INFO:tensorflow:['workertemp_2', 'workertemp_1', 'workertemp_3'] [worker-3]: I0520 05:20:46.482835 281473412068224 gce_failure_handler_test.py:203] ['workertemp_2', 'workertemp_1', 'workertemp_3'] [worker-3]: 2023-05-20 05:20:46.626519: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:699] Could not start gRPC server [worker-0]: Process _Process-2: [worker-0]: Traceback (most recent call last): [worker-0]: File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: self.run() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-0]: return self._actual_run() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-0]: app.run(lambda _: self._run_impl()) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-0]: _run_main(main, args) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-0]: sys.exit(main(argv)) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-0]: app.run(lambda _: self._run_impl()) [worker-0]: File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-0]: six.reraise(*info.exc_info) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-0]: raise value [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-0]: return_value = fn(*args, **kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 211, in worker_fn [worker-0]: self.assertNotEmpty(checkpoint_index) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 972, in assertNotEmpty [worker-0]: self.fail('{!r} has length of 0.'.format(container), msg) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 1814, in fail [worker-0]: return super(TestCase, self).fail(self._formatMessage(prefix, msg)) [worker-0]: File "/usr/lib/python3.10/unittest/case.py", line 675, in fail [worker-0]: raise self.failureException(msg) [worker-0]: AssertionError: [] has length of 0. [worker-1]: Exception in thread PeerTerminationWatcher-1: [worker-1]: Traceback (most recent call last): [worker-2]: Exception in thread PeerTerminationWatcher-2: [worker-2]: Traceback (most recent call last): [worker-1]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-2]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-1]: self.run() [worker-2]: self.run() [worker-1]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-2]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-2]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 1229, in _watch_step_to_save_key [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 1229, in _watch_step_to_save_key [worker-1]: step_value = context.context().get_config_key_value(_INITIAL_RUN_COUNT_KEY) [worker-2]: step_value = context.context().get_config_key_value(_INITIAL_RUN_COUNT_KEY) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 806, in get_config_key_value [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 806, in get_config_key_value [worker-1]: pywrap_tfe.TFE_GetConfigKeyValue(self._context_handle, key, [worker-2]: pywrap_tfe.TFE_GetConfigKeyValue(self._context_handle, key, [worker-1]: tensorflow.python.framework.errors_impl.UnavailableError: Socket closed [worker-2]: tensorflow.python.framework.errors_impl.UnavailableError: Socket closed [worker-1]: Additional GRPC error information from remote target /job:worker/replica:0/task:0 while calling /tensorflow.CoordinationService/GetKeyValue: [worker-2]: Additional GRPC error information from remote target /job:worker/replica:0/task:0 while calling /tensorflow.CoordinationService/GetKeyValue: [worker-1]: :{"created":"@1684560047.032965012","description":"Error received from peer ipv6:[::1]:24960","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed","grpc_status":14} [worker-2]: :{"created":"@1684560047.036206649","description":"Error received from peer ipv6:[::1]:24960","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed","grpc_status":14} [worker-3]: Exception in thread PeerTerminationWatcher-3: [worker-3]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-3]: self.run() [worker-3]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 1229, in _watch_step_to_save_key [worker-3]: step_value = context.context().get_config_key_value(_INITIAL_RUN_COUNT_KEY) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 806, in get_config_key_value [worker-3]: pywrap_tfe.TFE_GetConfigKeyValue(self._context_handle, key, [worker-3]: tensorflow.python.framework.errors_impl.UnavailableError: Socket closed [worker-3]: Additional GRPC error information from remote target /job:worker/replica:0/task:0 while calling /tensorflow.CoordinationService/GetKeyValue: [worker-3]: :{"created":"@1684560047.033374355","description":"Error received from peer ipv6:[::1]:24960","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed","grpc_status":14} [worker-2]: Process _Process-8: [worker-2]: Traceback (most recent call last): [worker-1]: Process _Process-3: [worker-1]: Traceback (most recent call last): [worker-1]: File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-2]: File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: self.run() [worker-2]: self.run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-1]: return self._actual_run() [worker-2]: return self._actual_run() [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-2]: app.run(lambda _: self._run_impl()) [worker-1]: app.run(lambda _: self._run_impl()) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-2]: _run_main(main, args) [worker-1]: _run_main(main, args) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-2]: sys.exit(main(argv)) [worker-1]: sys.exit(main(argv)) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-2]: app.run(lambda _: self._run_impl()) [worker-2]: File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-1]: app.run(lambda _: self._run_impl()) [worker-2]: self._target(*self._args, **self._kwargs) [worker-1]: File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-1]: self._target(*self._args, **self._kwargs) [worker-2]: six.reraise(*info.exc_info) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-1]: six.reraise(*info.exc_info) [worker-2]: raise value [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-1]: raise value [worker-2]: return_value = fn(*args, **kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 211, in worker_fn [worker-1]: return_value = fn(*args, **kwargs) [worker-2]: self.assertNotEmpty(checkpoint_index) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 211, in worker_fn [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 972, in assertNotEmpty [worker-1]: self.assertNotEmpty(checkpoint_index) [worker-2]: self.fail('{!r} has length of 0.'.format(container), msg) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 972, in assertNotEmpty [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 1814, in fail [worker-1]: self.fail('{!r} has length of 0.'.format(container), msg) [worker-2]: return super(TestCase, self).fail(self._formatMessage(prefix, msg)) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 1814, in fail [worker-2]: File "/usr/lib/python3.10/unittest/case.py", line 675, in fail [worker-1]: return super(TestCase, self).fail(self._formatMessage(prefix, msg)) [worker-2]: raise self.failureException(msg) [worker-1]: File "/usr/lib/python3.10/unittest/case.py", line 675, in fail [worker-2]: AssertionError: [] has length of 0. [worker-1]: raise self.failureException(msg) [worker-1]: AssertionError: [] has length of 0. [worker-3]: Process _Process-5: [worker-3]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-3]: self.run() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-3]: return self._actual_run() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-3]: app.run(lambda _: self._run_impl()) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-3]: _run_main(main, args) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-3]: sys.exit(main(argv)) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-3]: app.run(lambda _: self._run_impl()) [worker-3]: File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-3]: six.reraise(*info.exc_info) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-3]: raise value [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-3]: return_value = fn(*args, **kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 211, in worker_fn [worker-3]: self.assertNotEmpty(checkpoint_index) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 972, in assertNotEmpty [worker-3]: self.fail('{!r} has length of 0.'.format(container), msg) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 1814, in fail [worker-3]: return super(TestCase, self).fail(self._formatMessage(prefix, msg)) [worker-3]: File "/usr/lib/python3.10/unittest/case.py", line 675, in fail [worker-3]: raise self.failureException(msg) [worker-3]: AssertionError: [] has length of 0. [worker-3]: Process _Process-9: [worker-3]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-3]: self.run() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-3]: return self._actual_run() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-3]: app.run(lambda _: self._run_impl()) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-3]: _run_main(main, args) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-3]: sys.exit(main(argv)) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-3]: app.run(lambda _: self._run_impl()) [worker-3]: File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-3]: six.reraise(*info.exc_info) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-3]: raise value [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-3]: return_value = fn(*args, **kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 134, in worker_fn [worker-3]: strategy = collective_all_reduce_strategy.CollectiveAllReduceStrategy() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 186, in __init__ [worker-3]: CollectiveAllReduceExtended( [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 339, in __init__ [worker-3]: self._initialize_strategy(self._cluster_resolver, devices=devices) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 358, in _initialize_strategy [worker-3]: self._initialize_multi_worker(cluster_resolver) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 530, in _initialize_multi_worker [worker-3]: context.context().ensure_initialized() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized [worker-3]: pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) [worker-3]: tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server I0520 05:20:47.923911 281473279947648 multi_process_runner.py:646] worker-0 exit code: 1 I0520 05:20:47.924260 281473279947648 multi_process_runner.py:646] worker-1 exit code: 1 I0520 05:20:47.924396 281473279947648 multi_process_runner.py:646] worker-2 exit code: 1 I0520 05:20:47.924517 281473279947648 multi_process_runner.py:646] worker-3 exit code: 1 INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_basic_run_test_inputarg_manager_strategyoption_MWMSmultiworker): 40.02s I0520 05:20:48.078424 281473279947648 test_util.py:2464] time(__main__.GceFailureHandlingTest.test_basic_run_test_inputarg_manager_strategyoption_MWMSmultiworker): 40.02s [ FAILED ] GceFailureHandlingTest.test_basic_run_test_inputarg_manager_strategyoption_MWMSmultiworker [ RUN ] GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',) I0520 05:20:48.648740 281473279947648 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',) INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO I0520 05:20:48.649354 281473279947648 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO INFO:tensorflow:Start polling for termination signal. I0520 05:20:48.705391 281473279947648 failure_handling.py:683] Start polling for termination signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. I0520 05:20:48.706005 281473279947648 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. W0520 05:20:48.706333 281473279947648 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. INFO:tensorflow:Start training at 0 I0520 05:20:48.706507 281473279947648 gce_failure_handler_test.py:194] Start training at 0 WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffeee59b490> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0520 05:20:49.606645 281473279947648 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xfffeee59b490> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffeee4b8310> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0520 05:20:49.642389 281473279947648 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xfffeee4b8310> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. INFO:tensorflow:epoch 0 finished I0520 05:20:49.642917 281473279947648 gce_failure_handler_test.py:192] epoch 0 finished INFO:tensorflow:epoch 1 finished I0520 05:20:49.967316 281473279947648 gce_failure_handler_test.py:192] epoch 1 finished INFO:tensorflow:epoch 2 finished I0520 05:20:50.367481 281473279947648 gce_failure_handler_test.py:192] epoch 2 finished INFO:tensorflow:epoch 3 finished I0520 05:20:50.670246 281473279947648 gce_failure_handler_test.py:192] epoch 3 finished INFO:tensorflow:epoch 4 finished I0520 05:20:50.976741 281473279947648 gce_failure_handler_test.py:192] epoch 4 finished INFO:tensorflow:Training finished. I0520 05:20:50.977180 281473279947648 gce_failure_handler_test.py:244] Training finished. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker): 2.91s I0520 05:20:50.989460 281473279947648 test_util.py:2464] time(__main__.GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker): 2.91s [ OK ] GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker [ RUN ] GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',) I0520 05:20:51.001394 281473279947648 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',) INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO I0520 05:20:51.001878 281473279947648 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO INFO:tensorflow:Start polling for termination signal. I0520 05:20:51.047971 281473279947648 failure_handling.py:683] Start polling for termination signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. I0520 05:20:51.076224 281473279947648 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. INFO:tensorflow:Start training at 0 I0520 05:20:51.076743 281473279947648 gce_failure_handler_test.py:194] Start training at 0 INFO:tensorflow:epoch 0 finished I0520 05:20:51.382194 281473279947648 gce_failure_handler_test.py:192] epoch 0 finished INFO:tensorflow:epoch 1 finished I0520 05:20:51.998230 281473279947648 gce_failure_handler_test.py:192] epoch 1 finished INFO:tensorflow:epoch 2 finished I0520 05:20:52.436509 281473279947648 gce_failure_handler_test.py:192] epoch 2 finished INFO:tensorflow:epoch 3 finished I0520 05:20:52.846614 281473279947648 gce_failure_handler_test.py:192] epoch 3 finished INFO:tensorflow:epoch 4 finished I0520 05:20:53.176480 281473279947648 gce_failure_handler_test.py:192] epoch 4 finished INFO:tensorflow:Training finished. I0520 05:20:53.176922 281473279947648 gce_failure_handler_test.py:244] Training finished. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker): 2.19s I0520 05:20:53.182706 281473279947648 test_util.py:2464] time(__main__.GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker): 2.19s [ OK ] GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker [ RUN ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 17467 I0520 05:20:53.187864 281473279947648 test_util.py:3796] Using local port 17467 INFO:tensorflow:Using local port 21415 I0520 05:20:53.188384 281473279947648 test_util.py:3796] Using local port 21415 INFO:tensorflow:Using local port 19353 I0520 05:20:53.188798 281473279947648 test_util.py:3796] Using local port 19353 INFO:tensorflow:Using local port 19048 I0520 05:20:53.189199 281473279947648 test_util.py:3796] Using local port 19048 INFO:tensorflow:Cluster starting. I0520 05:20:53.236577 281473279947648 gce_failure_handler_test.py:405] Cluster starting. [worker-0]: I0520 05:20:53.276662 281473412068224 multi_process_runner.py:840] Subprocess with PID 526390 (worker, 0) is now being started. [worker-0]: I0520 05:20:53.277105 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:17467", "localhost:21415", "localhost:19353", "localhost:19048"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-0]: 2023-05-20 05:20:53.696758: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:17467 [worker-0]: 2023-05-20 05:20:53.746500: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 11593003455772030415 [worker-0]: 2023-05-20 05:20:53.747370: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-1]: I0520 05:20:53.827897 281473412068224 multi_process_runner.py:840] Subprocess with PID 526485 (worker, 1) is now being started. [worker-2]: I0520 05:20:53.911139 281473412068224 multi_process_runner.py:840] Subprocess with PID 526566 (worker, 2) is now being started. [worker-3]: I0520 05:20:53.911296 281473412068224 multi_process_runner.py:840] Subprocess with PID 526648 (worker, 3) is now being started. [worker-3]: I0520 05:20:53.911673 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:17467", "localhost:21415", "localhost:19353", "localhost:19048"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: I0520 05:20:53.911623 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:17467", "localhost:21415", "localhost:19353", "localhost:19048"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-1]: I0520 05:20:53.828362 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:17467", "localhost:21415", "localhost:19353", "localhost:19048"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: 2023-05-20 05:20:53.959791: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:19353 [worker-0]: 2023-05-20 05:20:53.963649: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 8199901483144391430 [worker-2]: 2023-05-20 05:20:53.963933: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: 2023-05-20 05:20:53.976549: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:19048 [worker-0]: 2023-05-20 05:20:53.983042: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 16204222082153459897 [worker-3]: 2023-05-20 05:20:53.983439: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-1]: 2023-05-20 05:20:54.167290: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:21415 [worker-0]: 2023-05-20 05:20:54.180882: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 14014489830371266078 [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: 2023-05-20 05:20:54.181708: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: I0520 05:20:54.185034 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0520 05:20:54.197052 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0520 05:20:54.216982 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0520 05:20:54.238754 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0520 05:20:54.360572 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0520 05:20:54.361105 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17467', 'localhost:21415', 'localhost:19353', 'localhost:19048']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0520 05:20:54.361314 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17467', 'localhost:21415', 'localhost:19353', 'localhost:19048']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0520 05:20:54.390068 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0520 05:20:54.391293 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17467', 'localhost:21415', 'localhost:19353', 'localhost:19048']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0520 05:20:54.391993 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17467', 'localhost:21415', 'localhost:19353', 'localhost:19048']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0520 05:20:54.460334 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0520 05:20:54.460939 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17467', 'localhost:21415', 'localhost:19353', 'localhost:19048']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0520 05:20:54.461153 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17467', 'localhost:21415', 'localhost:19353', 'localhost:19048']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0520 05:20:54.494416 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0520 05:20:54.494941 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17467', 'localhost:21415', 'localhost:19353', 'localhost:19048']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0520 05:20:54.495141 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17467', 'localhost:21415', 'localhost:19353', 'localhost:19048']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0520 05:20:54.670802 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0520 05:20:54.660226 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0520 05:20:54.692264 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0520 05:20:54.693911 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-2]: I0520 05:20:54.675879 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0520 05:20:54.686749 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-3]: I0520 05:20:54.706649 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-1]: Traceback (most recent call last): [worker-1]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-1]: self.run() [worker-1]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: if self._termination_watcher_fn(): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0520 05:20:54.708040 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0520 05:20:54.708527 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0520 05:20:54.708704 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0520 05:20:54.716302 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-0]: Traceback (most recent call last): [worker-0]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-0]: self.run() [worker-0]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0520 05:20:54.717177 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0520 05:20:54.717604 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0520 05:20:54.717778 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-3]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-3]: self.run() [worker-3]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: if self._termination_watcher_fn(): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0520 05:20:54.727804 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0520 05:20:54.728221 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0520 05:20:54.728389 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-2]: Traceback (most recent call last): [worker-2]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-2]: self.run() [worker-2]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-2]: self._target(*self._args, **self._kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: if self._termination_watcher_fn(): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0520 05:20:54.746198 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0520 05:20:54.746685 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0520 05:20:54.746868 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:54.916719 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:54.934560 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:54.944465 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:55.007813 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:55.131128 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:55.140177 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:55.170365 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:55.170444 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:55.360320 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:55.350739 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:55.371054 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:55.391071 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:55.603525 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:55.590838 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:55.620451 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:55.691527 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:55.842499 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:55.854888 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:55.860351 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:55.850891 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0520 05:20:55.952617 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5800d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0520 05:20:55.958553 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5800d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0520 05:20:55.947242 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0520 05:20:55.957794 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:55.980473 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:55.980776 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:56.001132 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:56.000419 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0520 05:20:56.066347 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0520 05:20:56.066782 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0520 05:20:56.071323 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0520 05:20:56.071699 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0520 05:20:56.076344 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0520 05:20:56.076716 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c580790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0520 05:20:56.079079 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c580790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0520 05:20:56.079501 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:56.080026 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:56.088859 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:56.100896 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:56.120394 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:56.220907 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:56.250400 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:56.260450 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:56.270549 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:56.390634 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:56.414816 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:56.450828 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:56.427715 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:56.565358 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:56.570514 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:56.580321 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:56.580932 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:56.671649 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:56.670554 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:56.731304 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:56.742844 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:56.910950 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:56.907278 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:57.013339 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:57.012573 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0520 05:20:57.098329 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0520 05:20:57.104263 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0520 05:20:57.109669 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0520 05:20:57.109494 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:57.151250 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:57.173547 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:57.191113 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:57.191340 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:57.268455 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:57.341589 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:57.440904 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:57.440661 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:57.541286 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:57.574549 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:57.567566 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:57.591016 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:57.771435 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:57.771377 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:57.782780 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:57.800935 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:57.950674 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:57.950957 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:57.971796 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:57.990672 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:58.196698 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:58.225871 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:58.237606 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:58.242863 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0520 05:20:58.608886 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0520 05:20:58.616182 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: I0520 05:20:58.615511 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0520 05:20:58.626698 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:58.641049 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:58.640827 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:58.641006 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:58.687011 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:58.865946 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:58.876887 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:58.891170 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:58.918081 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:59.132854 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:59.140522 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:59.165277 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:59.170556 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:59.469757 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:59.469814 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:59.469030 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:59.475640 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:59.531738 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:59.533258 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:59.551589 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:59.553620 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:20:59.692510 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:20:59.692457 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:20:59.712837 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:20:59.913313 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I0520 05:21:00.087291 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0520 05:21:00.096451 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-1]: I0520 05:21:00.097338 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: I0520 05:21:00.103140 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:00.107095 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:00.236853 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:00.409285 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:00.589508 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:00.700394 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:00.711122 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:00.730686 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:00.749906 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:01.030752 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:01.020526 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:01.050773 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:01.050318 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:01.259586 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:01.259804 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:01.327131 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:01.430876 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:01.643968 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:01.654955 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:01.650268 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:01.680072 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:01.971137 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:01.975265 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:01.980285 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:01.980593 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 4 finished [worker-0]: I0520 05:21:02.068606 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I0520 05:21:02.070724 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0520 05:21:02.072394 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:Training finished. [worker-1]: INFO:tensorflow:Training finished. [worker-2]: I0520 05:21:02.072710 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-1]: I0520 05:21:02.071004 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I0520 05:21:02.068926 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0520 05:21:02.086376 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0520 05:21:02.086674 281473412068224 gce_failure_handler_test.py:244] Training finished. INFO:tensorflow:Termination notice available. I0520 05:21:02.127450 281465430340064 gce_failure_handler_test.py:142] Termination notice available. INFO:tensorflow:Member single_worker has received termination notice. I0520 05:21:02.127861 281465430340064 failure_handling.py:701] Member single_worker has received termination notice. Exception ignored in: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 775, in __del__ self._stop_poll_termination_signal_thread() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 734, in _stop_poll_termination_signal_thread self._poll_termination_signal_thread.join() File "/usr/lib/python3.10/threading.py", line 1093, in join raise RuntimeError("cannot join current thread") RuntimeError: cannot join current thread INFO:tensorflow:restarting workers I0520 05:21:04.456506 281473279947648 gce_failure_handler_test.py:411] restarting workers INFO:tensorflow:workers restarted I0520 05:21:04.576236 281473279947648 gce_failure_handler_test.py:415] workers restarted [worker-0]: I0520 05:21:04.823725 281473412068224 multi_process_runner.py:840] Subprocess with PID 537545 (worker, 0) is now being started. [worker-3]: I0520 05:21:04.825542 281473412068224 multi_process_runner.py:840] Subprocess with PID 537565 (worker, 3) is now being started. [worker-0]: I0520 05:21:04.824177 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:17467", "localhost:21415", "localhost:19353", "localhost:19048"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0520 05:21:04.826705 281473412068224 multi_process_runner.py:840] Subprocess with PID 537558 (worker, 1) is now being started. [worker-3]: I0520 05:21:04.825955 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:17467", "localhost:21415", "localhost:19353", "localhost:19048"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-1]: I0520 05:21:04.827114 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:17467", "localhost:21415", "localhost:19353", "localhost:19048"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0520 05:21:04.826269 281473412068224 multi_process_runner.py:840] Subprocess with PID 537561 (worker, 2) is now being started. [worker-2]: I0520 05:21:04.826669 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:17467", "localhost:21415", "localhost:19353", "localhost:19048"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-1]: 2023-05-20 05:21:04.863022: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:21415 [worker-2]: 2023-05-20 05:21:04.862072: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:19353 [worker-0]: 2023-05-20 05:21:04.876629: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:17467 [worker-3]: 2023-05-20 05:21:04.883086: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:19048 [worker-0]: 2023-05-20 05:21:04.896433: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 9322923690217064501 [worker-1]: 2023-05-20 05:21:04.896750: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-2]: 2023-05-20 05:21:04.896757: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-05-20 05:21:04.896497: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 2748333715560179158 [worker-0]: 2023-05-20 05:21:04.896544: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 7830655349635707810 [worker-0]: 2023-05-20 05:21:04.896742: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: 2023-05-20 05:21:04.912975: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-05-20 05:21:04.912687: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 14676235013191788318 [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: I0520 05:21:04.914587 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0520 05:21:04.914660 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0520 05:21:04.914722 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0520 05:21:04.936974 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0520 05:21:04.978979 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0520 05:21:04.979465 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17467', 'localhost:21415', 'localhost:19353', 'localhost:19048']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0520 05:21:04.979662 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17467', 'localhost:21415', 'localhost:19353', 'localhost:19048']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0520 05:21:04.982040 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0520 05:21:04.982526 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17467', 'localhost:21415', 'localhost:19353', 'localhost:19048']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0520 05:21:04.982714 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17467', 'localhost:21415', 'localhost:19353', 'localhost:19048']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0520 05:21:04.978971 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0520 05:21:04.979760 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17467', 'localhost:21415', 'localhost:19353', 'localhost:19048']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0520 05:21:04.979951 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17467', 'localhost:21415', 'localhost:19353', 'localhost:19048']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0520 05:21:05.062009 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0520 05:21:05.062576 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17467', 'localhost:21415', 'localhost:19353', 'localhost:19048']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0520 05:21:05.062770 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17467', 'localhost:21415', 'localhost:19353', 'localhost:19048']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0520 05:21:05.210594 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0520 05:21:05.215026 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-3]: I0520 05:21:05.227363 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: Traceback (most recent call last): [worker-3]: I0520 05:21:05.240000 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0520 05:21:05.240790 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0520 05:21:05.242316 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: self.run() [worker-3]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: if self._termination_watcher_fn(): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0520 05:21:05.246716 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0520 05:21:05.247364 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: Traceback (most recent call last): [worker-2]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-2]: I0520 05:21:05.247754 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0520 05:21:05.248178 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0520 05:21:05.248353 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: self.run() [worker-2]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-2]: self._target(*self._args, **self._kwargs) [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: I0520 05:21:05.250049 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: if self._termination_watcher_fn(): [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: I0520 05:21:05.250857 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-1]: Traceback (most recent call last): [worker-1]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-1]: self.run() [worker-1]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: if self._termination_watcher_fn(): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0520 05:21:05.251237 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0520 05:21:05.253009 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0520 05:21:05.253174 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0520 05:21:05.316288 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-0]: Traceback (most recent call last): [worker-0]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-0]: self.run() [worker-0]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0520 05:21:05.337882 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0520 05:21:05.338312 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0520 05:21:05.338477 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:05.376552 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:05.373103 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:05.416776 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:05.577412 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:05.667803 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:05.663449 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:05.680266 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:05.688375 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:05.745012 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:05.745227 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:05.754144 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:05.770262 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:05.873962 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:05.880562 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:05.890108 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:05.890530 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:06.041560 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:06.050787 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:06.071164 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:06.090257 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0520 05:21:06.273497 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0520 05:21:06.273733 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0520 05:21:06.273697 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:06.282262 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0520 05:21:06.277885 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:06.288698 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:06.301114 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:06.305884 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0520 05:21:06.426469 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0520 05:21:06.426429 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0520 05:21:06.426413 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0520 05:21:06.434922 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: I0520 05:21:06.426869 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: INFO:tensorflow:epoch 0 finished [worker-3]: INFO:tensorflow:epoch 0 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 0 finished [worker-0]: I0520 05:21:06.426873 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: I0520 05:21:06.435300 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: I0520 05:21:06.426874 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: I0520 05:21:06.439930 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:06.460776 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:06.470399 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:06.471101 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:06.600576 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:06.610333 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:06.620546 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:06.612579 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:06.704742 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:06.706923 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:06.710682 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:06.751075 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:06.896311 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:06.900311 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:06.910667 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:06.920232 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:07.021969 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:07.024600 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:07.030470 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:07.059543 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:07.172256 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:07.172252 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:07.182807 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:07.200822 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0520 05:21:07.297563 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0520 05:21:07.302516 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: I0520 05:21:07.297465 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0520 05:21:07.307857 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:07.330544 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:07.330681 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:07.330346 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:07.350131 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:07.550595 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:07.540213 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:07.570451 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:07.570279 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:07.690117 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:07.720600 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:07.727248 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:07.710380 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:07.812757 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:07.841153 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:07.860537 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:07.888813 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:08.220330 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:08.220330 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:08.240265 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:08.250296 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:08.410622 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:08.440882 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:08.434060 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:08.450405 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0520 05:21:08.519357 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0520 05:21:08.526716 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0520 05:21:08.524187 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I0520 05:21:08.526440 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:08.535516 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:08.540147 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:08.570188 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:08.610501 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:08.775094 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:08.769669 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:08.790264 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:08.809128 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:08.896882 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:08.919359 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:08.917016 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:08.941626 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:09.040299 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:09.040311 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:09.026167 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:09.060237 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:09.243820 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:09.250423 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:09.274003 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:09.270632 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:09.346368 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:09.362370 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:09.382435 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:09.519999 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0520 05:21:09.619177 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I0520 05:21:09.624147 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I0520 05:21:09.635319 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: I0520 05:21:09.627316 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:09.640283 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:09.790203 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:09.810154 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:09.883819 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:09.971265 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:09.981716 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:10.010632 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:10.099903 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:10.151404 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:10.155908 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:10.170393 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:10.190248 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:10.263182 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:10.287362 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:10.280759 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:10.290357 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:10.371361 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:10.371141 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:10.371579 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:10.371886 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:10.421177 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:10.421712 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:10.439106 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:10.460841 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-0]: I0520 05:21:10.619967 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I0520 05:21:10.622493 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:Training finished. [worker-0]: INFO:tensorflow:Training finished. [worker-1]: I0520 05:21:10.622826 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-0]: I0520 05:21:10.620276 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-3]: I0520 05:21:10.618572 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0520 05:21:10.618886 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0520 05:21:10.674266 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I0520 05:21:10.674576 281473412068224 gce_failure_handler_test.py:244] Training finished. I0520 05:21:11.520200 281473279947648 multi_process_runner.py:646] worker-0 exit code: 0 I0520 05:21:11.520527 281473279947648 multi_process_runner.py:646] worker-1 exit code: 0 I0520 05:21:11.520668 281473279947648 multi_process_runner.py:646] worker-2 exit code: 0 I0520 05:21:11.520790 281473279947648 multi_process_runner.py:646] worker-3 exit code: 0 I0520 05:21:11.523484 281473279947648 multi_process_runner.py:662] Joining log reading threads. I0520 05:21:11.523738 281473279947648 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker): 18.5s I0520 05:21:11.690198 281473279947648 test_util.py:2464] time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker): 18.5s [ OK ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker [ RUN ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_7_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 24872 I0520 05:21:11.691871 281473279947648 test_util.py:3796] Using local port 24872 INFO:tensorflow:Using local port 21351 I0520 05:21:11.692313 281473279947648 test_util.py:3796] Using local port 21351 INFO:tensorflow:Using local port 19624 I0520 05:21:11.692723 281473279947648 test_util.py:3796] Using local port 19624 INFO:tensorflow:Using local port 18786 I0520 05:21:11.693120 281473279947648 test_util.py:3796] Using local port 18786 INFO:tensorflow:Cluster starting. I0520 05:21:11.742182 281473279947648 gce_failure_handler_test.py:405] Cluster starting. [worker-0]: I0520 05:21:11.782974 281473412068224 multi_process_runner.py:840] Subprocess with PID 551835 (worker, 0) is now being started. [worker-0]: I0520 05:21:11.783396 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24872", "localhost:21351", "localhost:19624", "localhost:18786"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-0]: 2023-05-20 05:21:11.846339: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24872 [worker-0]: 2023-05-20 05:21:11.862174: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 16820814662548895397 [worker-0]: 2023-05-20 05:21:11.862371: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: I0520 05:21:11.891031 281473412068224 multi_process_runner.py:840] Subprocess with PID 552226 (worker, 3) is now being started. [worker-3]: I0520 05:21:11.891482 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24872", "localhost:21351", "localhost:19624", "localhost:18786"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-1]: I0520 05:21:11.903188 281473412068224 multi_process_runner.py:840] Subprocess with PID 551879 (worker, 1) is now being started. [worker-2]: I0520 05:21:11.904375 281473412068224 multi_process_runner.py:840] Subprocess with PID 552031 (worker, 2) is now being started. [worker-1]: I0520 05:21:11.903623 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24872", "localhost:21351", "localhost:19624", "localhost:18786"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0520 05:21:11.904797 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24872", "localhost:21351", "localhost:19624", "localhost:18786"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-1]: 2023-05-20 05:21:11.996331: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:21351 [worker-0]: 2023-05-20 05:21:12.007206: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 9645662188407484954 [worker-1]: 2023-05-20 05:21:12.008145: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-2]: 2023-05-20 05:21:12.033982: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:19624 [worker-3]: 2023-05-20 05:21:12.047087: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:18786 [worker-0]: 2023-05-20 05:21:12.060415: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 16126840401740776661 [worker-3]: 2023-05-20 05:21:12.061251: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-05-20 05:21:12.144187: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 16326451822435585106 [worker-2]: 2023-05-20 05:21:12.156427: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0520 05:21:12.158757 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0520 05:21:12.158748 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0520 05:21:12.163018 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: I0520 05:21:12.160004 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0520 05:21:12.228958 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0520 05:21:12.229522 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24872', 'localhost:21351', 'localhost:19624', 'localhost:18786']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0520 05:21:12.229725 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24872', 'localhost:21351', 'localhost:19624', 'localhost:18786']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0520 05:21:12.233527 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0520 05:21:12.234031 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24872', 'localhost:21351', 'localhost:19624', 'localhost:18786']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0520 05:21:12.234230 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24872', 'localhost:21351', 'localhost:19624', 'localhost:18786']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0520 05:21:12.245911 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0520 05:21:12.246460 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24872', 'localhost:21351', 'localhost:19624', 'localhost:18786']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0520 05:21:12.246665 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24872', 'localhost:21351', 'localhost:19624', 'localhost:18786']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0520 05:21:12.491279 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0520 05:21:12.491783 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24872', 'localhost:21351', 'localhost:19624', 'localhost:18786']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0520 05:21:12.491988 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24872', 'localhost:21351', 'localhost:19624', 'localhost:18786']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0520 05:21:12.579636 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0520 05:21:12.727238 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0520 05:21:12.727356 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0520 05:21:12.730806 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0520 05:21:12.737661 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0520 05:21:12.738902 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-0]: I0520 05:21:12.739033 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-2]: Traceback (most recent call last): [worker-2]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-2]: self.run() [worker-2]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-2]: self._target(*self._args, **self._kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: if self._termination_watcher_fn(): [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-1]: Traceback (most recent call last): [worker-1]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-1]: self.run() [worker-1]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: if self._termination_watcher_fn(): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: I0520 05:21:12.741625 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: I0520 05:21:12.741185 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: W0520 05:21:12.742075 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-1]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0520 05:21:12.741523 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0520 05:21:12.742244 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0520 05:21:12.741684 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-3]: I0520 05:21:12.772445 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-0]: Traceback (most recent call last): [worker-0]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-3]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-3]: self.run() [worker-3]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-0]: I0520 05:21:12.756852 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: self._target(*self._args, **self._kwargs) [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: Instructions for updating: [worker-3]: if self._termination_watcher_fn(): [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: W0520 05:21:12.757272 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: Instructions for updating: [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0520 05:21:12.757435 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: self.run() [worker-0]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0520 05:21:12.793220 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0520 05:21:12.793718 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0520 05:21:12.793895 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:12.918422 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:12.921622 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:12.926339 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:13.028413 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:13.092971 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:13.102787 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:13.115115 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:13.130319 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:13.235442 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:13.235887 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:13.240121 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:13.247857 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:13.363310 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:13.372130 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:13.400872 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:13.444184 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:13.680944 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:13.670876 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:13.700683 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:13.756594 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0520 05:21:13.914506 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0520 05:21:13.915797 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0520 05:21:13.908947 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:13.917706 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:13.923221 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:13.924667 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c584040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0520 05:21:13.922663 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c584040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:13.950955 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0520 05:21:13.996321 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0520 05:21:13.996517 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-2]: W0520 05:21:13.996616 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: I0520 05:21:13.996745 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: I0520 05:21:13.996972 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0520 05:21:13.997082 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: I0520 05:21:14.005667 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:14.005497 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:14.005498 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: W0520 05:21:13.996180 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0520 05:21:13.996666 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:14.005458 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:14.132451 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:14.140252 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:14.177377 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:14.251876 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:14.335630 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:14.335925 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:14.343941 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:14.362215 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:14.431283 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:14.440635 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:14.455324 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:14.455242 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:14.546610 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:14.548765 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:14.554418 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:14.587249 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:14.661821 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:14.663422 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:14.669009 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:14.707380 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0520 05:21:14.776695 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0520 05:21:14.778152 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0520 05:21:14.778933 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:14.785641 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:14.800864 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0520 05:21:14.796626 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:14.795433 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:14.820881 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:14.918044 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:14.930879 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:14.930897 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:14.935226 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:15.016164 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:15.016376 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:15.016568 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:15.039734 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:15.095859 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:15.100158 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:15.096557 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:15.109330 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:15.196585 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:15.197792 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:15.200320 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:15.196602 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:15.252529 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:15.254259 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:15.252347 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:15.267225 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0520 05:21:15.343307 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I0520 05:21:15.356612 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0520 05:21:15.357214 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:15.365160 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:15.366092 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0520 05:21:15.366593 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:15.375926 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:15.370809 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:15.486952 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:15.488049 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:15.500256 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:15.514836 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:15.807641 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:15.824568 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:15.824939 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:15.858063 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:15.941026 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:15.952693 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:15.940742 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:15.960724 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:16.055794 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:16.060973 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:16.059049 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:16.093183 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:16.263669 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:16.287312 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:16.317330 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:16.364669 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0520 05:21:16.418502 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-2]: I0520 05:21:16.425004 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: I0520 05:21:16.425485 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I0520 05:21:16.431137 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:16.439393 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:16.571338 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:16.597371 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:16.757871 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:16.851034 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:16.859723 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:16.887445 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:16.915116 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:17.200771 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:17.200554 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:17.220936 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:17.221956 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:17.400954 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:17.401019 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:17.413418 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:17.421766 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:17.532830 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:17.551343 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:17.571100 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:17.571068 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:17.713787 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:17.718273 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:17.740826 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:17.750787 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-0]: I0520 05:21:17.839577 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: I0520 05:21:17.837681 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: INFO:tensorflow:Training finished. [worker-3]: INFO:tensorflow:Training finished. [worker-0]: I0520 05:21:17.839884 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-3]: I0520 05:21:17.838052 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0520 05:21:17.841816 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I0520 05:21:17.842165 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I0520 05:21:17.846594 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I0520 05:21:17.846938 281473412068224 gce_failure_handler_test.py:244] Training finished. INFO:tensorflow:restarting workers I0520 05:21:19.866525 281473279947648 gce_failure_handler_test.py:411] restarting workers INFO:tensorflow:workers restarted I0520 05:21:19.940423 281473279947648 gce_failure_handler_test.py:415] workers restarted [worker-0]: I0520 05:21:19.987614 281473412068224 multi_process_runner.py:840] Subprocess with PID 570130 (worker, 0) is now being started. [worker-0]: I0520 05:21:19.988019 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24872", "localhost:21351", "localhost:19624", "localhost:18786"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: I0520 05:21:19.992707 281473412068224 multi_process_runner.py:840] Subprocess with PID 570147 (worker, 2) is now being started. [worker-2]: I0520 05:21:19.993112 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24872", "localhost:21351", "localhost:19624", "localhost:18786"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-1]: I0520 05:21:19.997572 281473412068224 multi_process_runner.py:840] Subprocess with PID 570133 (worker, 1) is now being started. [worker-1]: I0520 05:21:19.997997 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24872", "localhost:21351", "localhost:19624", "localhost:18786"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-3]: I0520 05:21:20.011909 281473412068224 multi_process_runner.py:840] Subprocess with PID 570150 (worker, 3) is now being started. [worker-3]: I0520 05:21:20.012313 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24872", "localhost:21351", "localhost:19624", "localhost:18786"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-3]: 2023-05-20 05:21:20.084777: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:18786 [worker-1]: 2023-05-20 05:21:20.148575: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:21351 [worker-0]: 2023-05-20 05:21:20.258082: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24872 [worker-0]: 2023-05-20 05:21:20.312068: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 8333656327576592572 [worker-3]: 2023-05-20 05:21:20.312950: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-05-20 05:21:20.360909: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 4242991152545638836 [worker-1]: 2023-05-20 05:21:20.361709: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-05-20 05:21:20.367620: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 5705128768093143010 [worker-0]: 2023-05-20 05:21:20.367916: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-2]: 2023-05-20 05:21:20.450848: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:19624 [worker-0]: 2023-05-20 05:21:20.454014: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 11813879364364260230 [worker-2]: 2023-05-20 05:21:20.454620: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0520 05:21:20.457432 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0520 05:21:20.477390 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: I0520 05:21:20.458927 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0520 05:21:20.457319 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0520 05:21:20.604503 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0520 05:21:20.604961 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24872', 'localhost:21351', 'localhost:19624', 'localhost:18786']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0520 05:21:20.605160 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24872', 'localhost:21351', 'localhost:19624', 'localhost:18786']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0520 05:21:20.640945 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0520 05:21:20.641999 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24872', 'localhost:21351', 'localhost:19624', 'localhost:18786']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0520 05:21:20.642204 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24872', 'localhost:21351', 'localhost:19624', 'localhost:18786']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0520 05:21:20.698468 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0520 05:21:20.698938 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24872', 'localhost:21351', 'localhost:19624', 'localhost:18786']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0520 05:21:20.699140 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24872', 'localhost:21351', 'localhost:19624', 'localhost:18786']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0520 05:21:20.720204 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0520 05:21:20.720760 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24872', 'localhost:21351', 'localhost:19624', 'localhost:18786']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0520 05:21:20.720958 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24872', 'localhost:21351', 'localhost:19624', 'localhost:18786']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0520 05:21:20.909844 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0520 05:21:20.922371 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0520 05:21:20.936934 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0520 05:21:20.928536 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: I0520 05:21:20.942958 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-3]: I0520 05:21:20.944783 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-3]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-3]: self.run() [worker-3]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: if self._termination_watcher_fn(): [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0520 05:21:20.946667 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: I0520 05:21:20.947335 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0520 05:21:20.948451 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-1]: Traceback (most recent call last): [worker-1]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0520 05:21:20.948255 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0520 05:21:20.950053 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0520 05:21:20.950237 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: self.run() [worker-1]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: W0520 05:21:20.948803 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-2]: Traceback (most recent call last): [worker-2]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0520 05:21:20.955708 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0520 05:21:20.948962 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0520 05:21:20.956181 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0520 05:21:20.956358 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: if self._termination_watcher_fn(): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-2]: self.run() [worker-2]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-2]: self._target(*self._args, **self._kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: if self._termination_watcher_fn(): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: Traceback (most recent call last): [worker-0]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-0]: self.run() [worker-0]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0520 05:21:20.976245 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0520 05:21:20.976710 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0520 05:21:20.976884 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:21.158508 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:21.179066 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:21.196574 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:21.285231 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:21.386868 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:21.392112 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:21.420173 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:21.440649 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:21.610370 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:21.611644 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:21.620760 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:21.630488 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:21.750282 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:21.767892 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:21.770261 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:21.791897 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:21.930203 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:21.920117 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:21.941124 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:21.954423 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0520 05:21:22.087510 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0520 05:21:22.096967 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5800d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0520 05:21:22.101768 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5800d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0520 05:21:22.101888 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:22.110143 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:22.120230 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:22.140294 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:22.140664 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0520 05:21:22.212007 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c580790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0520 05:21:22.217794 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c580790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 0 finished [worker-0]: INFO:tensorflow:epoch 0 finished [worker-1]: I0520 05:21:22.218240 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0520 05:21:22.224163 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0520 05:21:22.224575 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: I0520 05:21:22.212486 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:22.233338 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0520 05:21:22.230314 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0520 05:21:22.230695 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:22.238407 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:22.239428 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:22.261372 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:22.390439 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:22.405853 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:22.430747 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:22.430482 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:22.560496 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:22.570138 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:22.690775 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:22.714577 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:22.831397 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:22.840395 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:22.842029 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:22.870497 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:22.980564 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:23.001310 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:23.020614 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:23.031142 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:23.130672 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:23.135613 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:23.152738 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:23.170717 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0520 05:21:23.265066 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: INFO:tensorflow:epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-1]: I0520 05:21:23.277576 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: I0520 05:21:23.282839 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: I0520 05:21:23.270296 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:23.296209 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:23.310380 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:23.311275 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:23.331872 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:23.449874 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:23.474248 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:23.473873 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:23.480291 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:23.582806 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:23.584418 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:23.585805 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:23.610607 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:23.690484 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:23.691633 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:23.710285 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:23.710538 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:23.816324 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:23.820488 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:23.827103 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:23.827431 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:23.897966 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:23.901090 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:23.907722 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:23.909656 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0520 05:21:23.967316 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0520 05:21:23.968508 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I0520 05:21:23.968404 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0520 05:21:23.986429 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: I0520 05:21:23.976997 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:23.983120 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:23.990384 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:24.010286 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:24.112965 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:24.108388 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:24.120504 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:24.121411 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:24.260328 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:24.255313 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:24.266287 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:24.280498 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:24.401491 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:24.421267 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:24.425862 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:24.418606 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:24.644707 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:24.647468 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:24.670471 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:24.660474 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:24.837180 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:24.843766 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:24.867694 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:25.128456 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0520 05:21:25.190173 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I0520 05:21:25.195670 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I0520 05:21:25.200645 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-1]: I0520 05:21:25.216639 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:25.250806 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:25.406098 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:25.421664 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:25.417038 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:25.473252 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:25.478114 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:25.473747 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:25.490374 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:25.564627 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:25.574135 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:25.574367 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:25.590155 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:25.693091 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:25.685285 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:25.690570 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:25.693299 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:25.790746 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:25.790984 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:25.790731 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:25.821551 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:25.919697 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:25.941167 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:25.941116 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:25.960724 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0520 05:21:26.041607 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0520 05:21:26.041931 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0520 05:21:26.044849 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I0520 05:21:26.045158 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-1]: INFO:tensorflow:epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-0]: I0520 05:21:26.058127 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I0520 05:21:26.058422 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-1]: I0520 05:21:26.056568 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I0520 05:21:26.056931 281473412068224 gce_failure_handler_test.py:244] Training finished. I0520 05:21:26.904207 281473279947648 multi_process_runner.py:646] worker-0 exit code: 0 I0520 05:21:26.904583 281473279947648 multi_process_runner.py:646] worker-1 exit code: 0 I0520 05:21:26.904728 281473279947648 multi_process_runner.py:646] worker-2 exit code: 0 I0520 05:21:26.904852 281473279947648 multi_process_runner.py:646] worker-3 exit code: 0 I0520 05:21:26.908358 281473279947648 multi_process_runner.py:662] Joining log reading threads. I0520 05:21:26.908713 281473279947648 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_7_inputarg_manager_strategyoption_MWMSmultiworker): 15.54s I0520 05:21:27.226672 281473279947648 test_util.py:2464] time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_7_inputarg_manager_strategyoption_MWMSmultiworker): 15.54s [ OK ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_7_inputarg_manager_strategyoption_MWMSmultiworker [ RUN ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 21956 I0520 05:21:27.228339 281473279947648 test_util.py:3796] Using local port 21956 INFO:tensorflow:Using local port 22620 I0520 05:21:27.228776 281473279947648 test_util.py:3796] Using local port 22620 INFO:tensorflow:Using local port 15563 I0520 05:21:27.229183 281473279947648 test_util.py:3796] Using local port 15563 INFO:tensorflow:Using local port 21884 I0520 05:21:27.229579 281473279947648 test_util.py:3796] Using local port 21884 INFO:tensorflow:Cluster starting. I0520 05:21:27.582203 281473279947648 gce_failure_handler_test.py:405] Cluster starting. [worker-0]: I0520 05:21:28.210438 281473412068224 multi_process_runner.py:840] Subprocess with PID 581255 (worker, 0) is now being started. [worker-1]: I0520 05:21:28.233981 281473412068224 multi_process_runner.py:840] Subprocess with PID 581259 (worker, 1) is now being started. [worker-0]: I0520 05:21:28.210830 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:21956", "localhost:22620", "localhost:15563", "localhost:21884"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0520 05:21:28.234372 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:21956", "localhost:22620", "localhost:15563", "localhost:21884"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-3]: I0520 05:21:28.238893 281473412068224 multi_process_runner.py:840] Subprocess with PID 581270 (worker, 3) is now being started. [worker-3]: I0520 05:21:28.239274 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:21956", "localhost:22620", "localhost:15563", "localhost:21884"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: I0520 05:21:28.256670 281473412068224 multi_process_runner.py:840] Subprocess with PID 581263 (worker, 2) is now being started. [worker-2]: I0520 05:21:28.257047 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:21956", "localhost:22620", "localhost:15563", "localhost:21884"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-2]: 2023-05-20 05:21:28.312445: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:15563 [worker-3]: 2023-05-20 05:21:28.328494: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:21884 [worker-0]: 2023-05-20 05:21:28.363541: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:21956 [worker-1]: 2023-05-20 05:21:28.368641: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:22620 [worker-0]: 2023-05-20 05:21:28.376850: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 12924384342677568405 [worker-0]: 2023-05-20 05:21:28.376936: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 12329230860511628307 [worker-2]: 2023-05-20 05:21:28.377214: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-05-20 05:21:28.377213: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: 2023-05-20 05:21:28.377981: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-05-20 05:21:28.377802: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 9516856281548675773 [worker-0]: 2023-05-20 05:21:28.421806: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 13228271496314218692 [worker-1]: 2023-05-20 05:21:28.422485: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0520 05:21:28.424634 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0520 05:21:28.437450 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0520 05:21:28.455303 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0520 05:21:28.449279 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0520 05:21:28.504029 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0520 05:21:28.504465 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21956', 'localhost:22620', 'localhost:15563', 'localhost:21884']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0520 05:21:28.504656 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21956', 'localhost:22620', 'localhost:15563', 'localhost:21884']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0520 05:21:28.509438 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0520 05:21:28.509915 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21956', 'localhost:22620', 'localhost:15563', 'localhost:21884']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0520 05:21:28.510112 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21956', 'localhost:22620', 'localhost:15563', 'localhost:21884']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0520 05:21:28.513859 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0520 05:21:28.514783 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21956', 'localhost:22620', 'localhost:15563', 'localhost:21884']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0520 05:21:28.514991 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21956', 'localhost:22620', 'localhost:15563', 'localhost:21884']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0520 05:21:28.515589 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0520 05:21:28.515985 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21956', 'localhost:22620', 'localhost:15563', 'localhost:21884']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0520 05:21:28.516210 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21956', 'localhost:22620', 'localhost:15563', 'localhost:21884']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0520 05:21:28.575527 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: I0520 05:21:28.575729 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: I0520 05:21:28.575955 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0520 05:21:28.583910 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: Traceback (most recent call last): [worker-0]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-0]: I0520 05:21:28.584506 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0520 05:21:28.584979 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0520 05:21:28.585161 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: self.run() [worker-0]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0520 05:21:28.590462 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0520 05:21:28.591289 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0520 05:21:28.591800 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0520 05:21:28.592210 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0520 05:21:28.592384 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: Traceback (most recent call last): [worker-2]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-2]: self.run() [worker-2]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-2]: self._target(*self._args, **self._kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: if self._termination_watcher_fn(): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0520 05:21:28.607000 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-3]: I0520 05:21:28.616664 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-3]: I0520 05:21:28.617283 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0520 05:21:28.617797 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0520 05:21:28.618044 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-1]: Traceback (most recent call last): [worker-3]: self.run() [worker-3]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: if self._termination_watcher_fn(): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-1]: self.run() [worker-1]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: if self._termination_watcher_fn(): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0520 05:21:28.627924 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0520 05:21:28.628305 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0520 05:21:28.628471 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:28.681787 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:28.696208 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:28.715304 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:28.797250 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:28.872989 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:28.880020 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:28.873440 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:28.915005 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:29.004985 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:29.007408 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:29.003491 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:29.007533 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:29.087584 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:29.088363 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:29.087614 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:29.101289 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:29.157559 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:29.158651 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:29.175417 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:29.181415 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0520 05:21:29.237495 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0520 05:21:29.237627 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0520 05:21:29.244969 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:29.245470 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0520 05:21:29.245159 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:29.252806 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:29.268022 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:29.271098 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0520 05:21:29.337818 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0520 05:21:29.338162 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0520 05:21:29.339634 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0520 05:21:29.339978 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:29.346136 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0520 05:21:29.357264 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0520 05:21:29.357633 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:29.360379 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0520 05:21:29.357291 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0520 05:21:29.357652 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:29.365384 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:29.365400 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:29.422988 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:29.427580 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:29.422905 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:29.460326 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:29.515430 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:29.520211 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:29.524393 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:29.530317 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:29.602263 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:29.602262 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:29.602580 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:29.614276 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:29.665444 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:29.665658 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:29.668612 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:29.681933 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:29.782818 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:29.789013 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:29.812013 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:29.822159 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0520 05:21:29.897243 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:29.905138 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0520 05:21:29.906399 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0520 05:21:29.908610 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:29.915113 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0520 05:21:29.920250 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:29.917977 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:29.936995 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:30.031450 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:30.034546 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:30.069732 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:30.070820 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:30.129273 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:30.133708 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:30.149298 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:30.150228 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:30.218186 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:30.234956 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:30.240613 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:30.270190 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:30.374773 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:30.393586 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:30.402481 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:30.404045 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:30.521453 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:30.540632 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:30.541514 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:30.550220 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I0520 05:21:30.609336 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:30.617511 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0520 05:21:30.615497 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0520 05:21:30.619978 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-2]: I0520 05:21:30.628742 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:30.624947 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:30.632942 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:30.650946 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:30.709041 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:30.708960 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:30.712977 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:30.711117 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:30.766382 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:30.769440 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:30.780263 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:30.780346 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:30.874904 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:30.876353 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:30.882768 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:30.930308 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:31.016039 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:31.034697 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:31.067024 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:31.086959 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:31.186858 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:31.186525 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:31.202114 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:31.347081 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0520 05:21:31.396823 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I0520 05:21:31.396773 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I0520 05:21:31.406578 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-1]: I0520 05:21:31.410974 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:31.430333 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:31.504452 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:31.507423 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:31.528979 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:31.585171 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:31.581319 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:31.618579 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:31.640573 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:31.750533 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:31.750788 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:31.780176 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:31.790306 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:31.880198 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:31.900357 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:31.890280 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:31.890254 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:32.004341 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:32.015979 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:32.020678 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:32.060297 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:32.125601 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:32.140255 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:32.160297 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:32.150343 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0520 05:21:32.226409 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0520 05:21:32.226714 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I0520 05:21:32.231540 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I0520 05:21:32.231849 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0520 05:21:32.239655 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:Training finished. [worker-0]: INFO:tensorflow:epoch 4 finished [worker-0]: I0520 05:21:32.241347 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I0520 05:21:32.241630 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-2]: I0520 05:21:32.239965 281473412068224 gce_failure_handler_test.py:244] Training finished. INFO:tensorflow:restarting workers I0520 05:21:34.066883 281473279947648 gce_failure_handler_test.py:411] restarting workers INFO:tensorflow:workers restarted I0520 05:21:34.146504 281473279947648 gce_failure_handler_test.py:415] workers restarted [worker-1]: I0520 05:21:34.557723 281473412068224 multi_process_runner.py:840] Subprocess with PID 588259 (worker, 1) is now being started. [worker-0]: I0520 05:21:34.582316 281473412068224 multi_process_runner.py:840] Subprocess with PID 588256 (worker, 0) is now being started. [worker-1]: I0520 05:21:34.558103 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:21956", "localhost:22620", "localhost:15563", "localhost:21884"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: I0520 05:21:34.582666 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:21956", "localhost:22620", "localhost:15563", "localhost:21884"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-3]: I0520 05:21:34.622503 281473412068224 multi_process_runner.py:840] Subprocess with PID 588324 (worker, 3) is now being started. [worker-3]: I0520 05:21:34.622883 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:21956", "localhost:22620", "localhost:15563", "localhost:21884"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: I0520 05:21:34.628721 281473412068224 multi_process_runner.py:840] Subprocess with PID 588291 (worker, 2) is now being started. [worker-0]: 2023-05-20 05:21:34.649071: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:21956 [worker-3]: 2023-05-20 05:21:34.663984: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:21884 [worker-2]: I0520 05:21:34.629096 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:21956", "localhost:22620", "localhost:15563", "localhost:21884"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-0]: 2023-05-20 05:21:34.676694: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 14789974963939523932 [worker-0]: 2023-05-20 05:21:34.687628: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: 2023-05-20 05:21:34.688360: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-05-20 05:21:34.688056: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 10538952009392310098 [worker-2]: 2023-05-20 05:21:34.697201: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:15563 [worker-0]: 2023-05-20 05:21:34.727153: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 1046487813145872404 [worker-2]: 2023-05-20 05:21:34.727892: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-1]: 2023-05-20 05:21:34.916805: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:22620 [worker-0]: 2023-05-20 05:21:34.951518: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 15397753072366817674 [worker-1]: 2023-05-20 05:21:34.976172: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0520 05:21:34.989753 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0520 05:21:34.978390 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0520 05:21:35.007103 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-1]: I0520 05:21:34.986762 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0520 05:21:35.036978 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0520 05:21:35.037417 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21956', 'localhost:22620', 'localhost:15563', 'localhost:21884']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0520 05:21:35.037612 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21956', 'localhost:22620', 'localhost:15563', 'localhost:21884']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0520 05:21:35.105094 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0520 05:21:35.105523 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21956', 'localhost:22620', 'localhost:15563', 'localhost:21884']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0520 05:21:35.105712 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21956', 'localhost:22620', 'localhost:15563', 'localhost:21884']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0520 05:21:35.135944 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0520 05:21:35.137247 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21956', 'localhost:22620', 'localhost:15563', 'localhost:21884']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0520 05:21:35.137943 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21956', 'localhost:22620', 'localhost:15563', 'localhost:21884']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0520 05:21:35.157915 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0520 05:21:35.158359 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21956', 'localhost:22620', 'localhost:15563', 'localhost:21884']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0520 05:21:35.158554 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21956', 'localhost:22620', 'localhost:15563', 'localhost:21884']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0520 05:21:35.244888 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0520 05:21:35.246845 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0520 05:21:35.256540 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: I0520 05:21:35.256458 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-0]: Traceback (most recent call last): [worker-0]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0520 05:21:35.257110 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0520 05:21:35.257420 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0520 05:21:35.257590 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: self.run() [worker-0]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0520 05:21:35.266249 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0520 05:21:35.276250 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0520 05:21:35.277593 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-3]: I0520 05:21:35.278789 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-3]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-3]: self.run() [worker-3]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: if self._termination_watcher_fn(): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0520 05:21:35.281023 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0520 05:21:35.281300 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0520 05:21:35.281453 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-2]: Traceback (most recent call last): [worker-2]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-2]: self.run() [worker-2]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-2]: self._target(*self._args, **self._kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: if self._termination_watcher_fn(): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0520 05:21:35.288281 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0520 05:21:35.288651 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0520 05:21:35.288818 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-1]: Traceback (most recent call last): [worker-1]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-1]: self.run() [worker-1]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: if self._termination_watcher_fn(): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0520 05:21:35.306180 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0520 05:21:35.306568 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0520 05:21:35.306731 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:35.393335 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:35.430061 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:35.535800 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:35.544747 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:35.661231 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:35.667983 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:35.671506 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:35.679873 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:35.805385 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:35.813955 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:35.820260 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:35.830101 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:35.960836 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:35.972787 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:35.971188 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:36.002321 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:36.085027 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:36.086689 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:36.085420 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:36.118198 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0520 05:21:36.199282 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0520 05:21:36.207726 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5800d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0520 05:21:36.212503 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5800d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:36.233334 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: W0520 05:21:36.212855 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:36.223039 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:36.220526 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:36.257014 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0520 05:21:36.318831 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0520 05:21:36.319186 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0520 05:21:36.327781 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0520 05:21:36.328124 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:36.326530 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c580790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0520 05:21:36.336323 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c580790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 0 finished [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: I0520 05:21:36.336666 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: W0520 05:21:36.337066 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0520 05:21:36.337409 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:36.344705 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:36.351777 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:36.350285 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:36.442767 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:36.442813 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:36.447859 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:36.454632 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:36.574319 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:36.580125 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:36.590071 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:36.590075 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:36.700365 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:36.707267 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:36.700303 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:36.722649 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:36.819716 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:36.821996 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:36.832106 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:36.860154 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:36.910553 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:36.910767 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:36.941896 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:36.940224 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0520 05:21:37.051824 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: I0520 05:21:37.046670 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0520 05:21:37.057770 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:37.059760 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0520 05:21:37.062976 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:37.090271 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:37.100202 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:37.121138 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:37.285295 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:37.290912 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:37.311171 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:37.330256 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:37.450158 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:37.441968 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:37.497138 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:37.500121 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:37.609878 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:37.619802 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:37.619694 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:37.632666 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:37.754032 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:37.750104 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:37.770237 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:37.760157 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:37.847930 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:37.870131 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:37.870171 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:37.887054 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I0520 05:21:37.982417 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-3]: INFO:tensorflow:epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0520 05:21:37.987998 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0520 05:21:37.990921 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-3]: I0520 05:21:37.986382 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:38.010080 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:38.010077 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:38.042040 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:38.028832 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:38.118073 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:38.124508 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:38.130902 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:38.162123 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:38.320136 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:38.340291 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:38.350121 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:38.366938 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:38.480156 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:38.480081 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:38.510343 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:38.473609 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:38.602499 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:38.615161 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:38.610190 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:38.625088 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:38.729744 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:38.812167 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:38.888178 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:38.935190 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 3 finished [worker-1]: I0520 05:21:38.999523 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I0520 05:21:39.004116 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-3]: INFO:tensorflow:epoch 3 finished [worker-0]: I0520 05:21:39.006553 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: I0520 05:21:39.010988 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:39.020356 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:39.050760 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:39.222491 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:39.238721 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:39.307030 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:39.320368 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:39.304312 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:39.311362 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:39.411881 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:39.416999 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:39.431408 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:39.420267 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:39.590400 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:39.599339 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:39.602385 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:39.660598 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:39.813929 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:39.827232 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:39.866972 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:39.873956 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:39.950550 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:39.952322 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:39.955413 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:39.962593 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0520 05:21:40.079358 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: I0520 05:21:40.077123 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:Training finished. [worker-0]: INFO:tensorflow:Training finished. [worker-1]: I0520 05:21:40.079015 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: I0520 05:21:40.077439 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-2]: I0520 05:21:40.079627 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I0520 05:21:40.079273 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0520 05:21:40.077867 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0520 05:21:40.078143 281473412068224 gce_failure_handler_test.py:244] Training finished. I0520 05:21:41.096647 281473279947648 multi_process_runner.py:646] worker-0 exit code: 0 I0520 05:21:41.097003 281473279947648 multi_process_runner.py:646] worker-1 exit code: 0 I0520 05:21:41.097142 281473279947648 multi_process_runner.py:646] worker-2 exit code: 0 I0520 05:21:41.097265 281473279947648 multi_process_runner.py:646] worker-3 exit code: 0 I0520 05:21:41.099949 281473279947648 multi_process_runner.py:662] Joining log reading threads. I0520 05:21:41.100183 281473279947648 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker): 14.07s I0520 05:21:41.301942 281473279947648 test_util.py:2464] time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker): 14.07s [ OK ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker [ RUN ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_7_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 24821 I0520 05:21:41.303961 281473279947648 test_util.py:3796] Using local port 24821 INFO:tensorflow:Using local port 23775 I0520 05:21:41.304434 281473279947648 test_util.py:3796] Using local port 23775 INFO:tensorflow:Using local port 17136 I0520 05:21:41.304845 281473279947648 test_util.py:3796] Using local port 17136 INFO:tensorflow:Using local port 23864 I0520 05:21:41.305240 281473279947648 test_util.py:3796] Using local port 23864 INFO:tensorflow:Cluster starting. I0520 05:21:41.777165 281473279947648 gce_failure_handler_test.py:405] Cluster starting. [worker-0]: I0520 05:21:41.847500 281473412068224 multi_process_runner.py:840] Subprocess with PID 597329 (worker, 0) is now being started. [worker-0]: I0520 05:21:41.847880 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24821", "localhost:23775", "localhost:17136", "localhost:23864"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-0]: 2023-05-20 05:21:42.036282: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24821 [worker-2]: I0520 05:21:42.047437 281473412068224 multi_process_runner.py:840] Subprocess with PID 597393 (worker, 2) is now being started. [worker-1]: I0520 05:21:42.057803 281473412068224 multi_process_runner.py:840] Subprocess with PID 597372 (worker, 1) is now being started. [worker-0]: 2023-05-20 05:21:42.073964: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 10862072677469084389 [worker-0]: 2023-05-20 05:21:42.079802: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-2]: I0520 05:21:42.047816 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24821", "localhost:23775", "localhost:17136", "localhost:23864"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-1]: I0520 05:21:42.058181 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24821", "localhost:23775", "localhost:17136", "localhost:23864"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-3]: I0520 05:21:42.127693 281473412068224 multi_process_runner.py:840] Subprocess with PID 597399 (worker, 3) is now being started. [worker-3]: I0520 05:21:42.128063 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24821", "localhost:23775", "localhost:17136", "localhost:23864"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-3]: 2023-05-20 05:21:42.266748: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:23864 [worker-1]: 2023-05-20 05:21:42.300529: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:23775 [worker-2]: 2023-05-20 05:21:42.314954: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:17136 [worker-0]: 2023-05-20 05:21:42.316970: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 1456793245783657498 [worker-1]: 2023-05-20 05:21:42.317642: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-05-20 05:21:42.372694: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 15782896929136360285 [worker-3]: 2023-05-20 05:21:42.373567: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-05-20 05:21:42.406182: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 8752545363321049726 [worker-2]: 2023-05-20 05:21:42.416403: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0520 05:21:42.425103 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0520 05:21:42.426262 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0520 05:21:42.436791 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0520 05:21:42.456746 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0520 05:21:42.526243 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0520 05:21:42.528584 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24821', 'localhost:23775', 'localhost:17136', 'localhost:23864']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0520 05:21:42.528792 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24821', 'localhost:23775', 'localhost:17136', 'localhost:23864']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0520 05:21:42.579248 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0520 05:21:42.580445 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24821', 'localhost:23775', 'localhost:17136', 'localhost:23864']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0520 05:21:42.581098 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24821', 'localhost:23775', 'localhost:17136', 'localhost:23864']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0520 05:21:42.623976 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0520 05:21:42.624402 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24821', 'localhost:23775', 'localhost:17136', 'localhost:23864']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0520 05:21:42.624595 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24821', 'localhost:23775', 'localhost:17136', 'localhost:23864']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0520 05:21:42.747897 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0520 05:21:42.748341 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24821', 'localhost:23775', 'localhost:17136', 'localhost:23864']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0520 05:21:42.748540 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24821', 'localhost:23775', 'localhost:17136', 'localhost:23864']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0520 05:21:42.869260 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0520 05:21:42.870312 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0520 05:21:42.870948 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-1]: Traceback (most recent call last): [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0520 05:21:42.871424 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: I0520 05:21:42.871569 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0520 05:21:42.871788 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0520 05:21:42.871982 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: I0520 05:21:42.875988 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: self.run() [worker-1]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: if self._termination_watcher_fn(): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0520 05:21:42.883664 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0520 05:21:42.886981 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-0]: Traceback (most recent call last): [worker-0]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-0]: self.run() [worker-0]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0520 05:21:42.889961 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0520 05:21:42.890327 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0520 05:21:42.890492 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-2]: Traceback (most recent call last): [worker-2]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-2]: self.run() [worker-2]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-2]: self._target(*self._args, **self._kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: if self._termination_watcher_fn(): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0520 05:21:42.907911 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0520 05:21:42.908283 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0520 05:21:42.908446 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-3]: I0520 05:21:42.936713 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-3]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-3]: self.run() [worker-3]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0520 05:21:42.938178 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0520 05:21:42.938508 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0520 05:21:42.938665 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: if self._termination_watcher_fn(): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:43.055444 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:43.085778 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:43.104549 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:43.143393 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:43.248270 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:43.270425 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:43.260204 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:43.309914 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:43.401651 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:43.398872 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:43.412092 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:43.427119 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:43.480232 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:43.478906 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:43.484805 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:43.491437 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:43.591042 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:43.610309 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:43.620892 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:43.617847 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0520 05:21:43.709185 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0520 05:21:43.707993 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0520 05:21:43.716972 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:43.715821 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0520 05:21:43.740220 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:43.738300 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:43.717452 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:43.760206 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0520 05:21:43.847010 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0520 05:21:43.847334 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: W0520 05:21:43.841836 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0520 05:21:43.842184 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0520 05:21:43.852806 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0520 05:21:43.853134 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: W0520 05:21:43.854917 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0520 05:21:43.855250 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: I0520 05:21:43.864445 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:43.882116 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:43.872804 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:43.890336 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:44.040238 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:44.050183 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:44.070164 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:44.080269 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:44.181191 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:44.230268 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:44.223704 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:44.237848 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:44.434424 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:44.430093 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:44.420209 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:44.451385 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:44.540274 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:44.560330 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:44.551960 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:44.553041 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:44.690185 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:44.698049 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:44.711755 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:44.732480 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0520 05:21:44.829084 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0520 05:21:44.833589 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0520 05:21:44.833952 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0520 05:21:44.838371 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:44.850174 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:44.860849 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:44.862848 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:44.880309 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:44.988463 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:44.988473 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:45.010177 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:45.011034 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:45.105179 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:45.112705 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:45.111854 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:45.120129 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:45.263897 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:45.262904 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:45.270354 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:45.279646 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:45.514819 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:45.530643 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:45.530045 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:45.666954 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:45.810259 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:45.880205 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:45.890052 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:46.040183 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0520 05:21:46.136761 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: I0520 05:21:46.136206 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0520 05:21:46.140931 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0520 05:21:46.146523 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:46.187199 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:46.180147 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:46.190080 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:46.230172 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:46.350246 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:46.388976 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:46.377942 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:46.377647 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:46.473971 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:46.477291 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:46.467229 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:46.492918 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:46.640009 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:46.650115 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:46.680520 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:46.670526 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:46.845771 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:46.871741 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:46.860282 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:46.880166 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:46.982169 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:46.972264 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:46.988027 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:47.146931 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0520 05:21:47.229250 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I0520 05:21:47.235067 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-1]: I0520 05:21:47.240527 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I0520 05:21:47.256389 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:47.261823 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:47.458082 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:47.469101 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:47.488667 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:47.671018 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:47.698830 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:47.726387 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:47.730085 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:47.900181 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:47.924032 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:47.960091 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:47.990217 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:48.110010 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:48.112020 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:48.130277 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:48.127242 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:48.264358 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:48.270065 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:48.282522 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:48.280268 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:48.344832 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:48.356000 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:48.370182 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:48.390778 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0520 05:21:48.508592 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0520 05:21:48.508892 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-0]: INFO:tensorflow:epoch 4 finished [worker-0]: I0520 05:21:48.511095 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0520 05:21:48.512956 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: INFO:tensorflow:Training finished. [worker-2]: INFO:tensorflow:Training finished. [worker-0]: I0520 05:21:48.511371 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-2]: I0520 05:21:48.513229 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I0520 05:21:48.521472 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I0520 05:21:48.521761 281473412068224 gce_failure_handler_test.py:244] Training finished. INFO:tensorflow:restarting workers I0520 05:21:49.916629 281473279947648 gce_failure_handler_test.py:411] restarting workers [worker-0]: I0520 05:21:50.267361 281473412068224 multi_process_runner.py:840] Subprocess with PID 609736 (worker, 0) is now being started. [worker-0]: I0520 05:21:50.267736 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24821", "localhost:23775", "localhost:17136", "localhost:23864"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0520 05:21:50.417698 281473412068224 multi_process_runner.py:840] Subprocess with PID 610382 (worker, 1) is now being started. INFO:tensorflow:workers restarted I0520 05:21:50.455001 281473279947648 gce_failure_handler_test.py:415] workers restarted [worker-1]: I0520 05:21:50.418166 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24821", "localhost:23775", "localhost:17136", "localhost:23864"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0520 05:21:50.471354 281473412068224 multi_process_runner.py:840] Subprocess with PID 610930 (worker, 2) is now being started. [worker-2]: I0520 05:21:50.471727 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24821", "localhost:23775", "localhost:17136", "localhost:23864"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-0]: 2023-05-20 05:21:50.504127: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24821 [worker-0]: 2023-05-20 05:21:50.528415: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 8674533623238517812 [worker-0]: 2023-05-20 05:21:50.529294: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: I0520 05:21:50.537957 281473412068224 multi_process_runner.py:840] Subprocess with PID 610934 (worker, 3) is now being started. [worker-3]: I0520 05:21:50.538319 281473412068224 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24821", "localhost:23775", "localhost:17136", "localhost:23864"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: 2023-05-20 05:21:50.566143: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:17136 [worker-3]: 2023-05-20 05:21:50.633918: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:23864 [worker-0]: 2023-05-20 05:21:50.661629: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 3361312930094177394 [worker-2]: 2023-05-20 05:21:50.665920: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-1]: 2023-05-20 05:21:50.727191: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:23775 [worker-0]: 2023-05-20 05:21:50.730271: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 11894359182214849057 [worker-3]: 2023-05-20 05:21:50.730582: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-05-20 05:21:50.730849: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 4393903250248691827 [worker-1]: 2023-05-20 05:21:50.731267: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: I0520 05:21:50.735053 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0520 05:21:50.749424 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: I0520 05:21:50.733937 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0520 05:21:50.777634 281473412068224 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0520 05:21:50.880812 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0520 05:21:50.882199 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24821', 'localhost:23775', 'localhost:17136', 'localhost:23864']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0520 05:21:50.882866 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24821', 'localhost:23775', 'localhost:17136', 'localhost:23864']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0520 05:21:50.981263 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0520 05:21:50.981783 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24821', 'localhost:23775', 'localhost:17136', 'localhost:23864']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0520 05:21:50.981975 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24821', 'localhost:23775', 'localhost:17136', 'localhost:23864']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0520 05:21:51.101120 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0520 05:21:51.102623 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24821', 'localhost:23775', 'localhost:17136', 'localhost:23864']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0520 05:21:51.103307 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24821', 'localhost:23775', 'localhost:17136', 'localhost:23864']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0520 05:21:50.970302 281473412068224 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0520 05:21:51.126944 281473412068224 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24821', 'localhost:23775', 'localhost:17136', 'localhost:23864']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0520 05:21:51.127225 281473412068224 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24821', 'localhost:23775', 'localhost:17136', 'localhost:23864']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0520 05:21:51.250412 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0520 05:21:51.252576 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0520 05:21:51.266309 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0520 05:21:51.267343 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0520 05:21:51.272792 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-0]: Traceback (most recent call last): [worker-0]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-0]: self.run() [worker-0]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0520 05:21:51.275861 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0520 05:21:51.276273 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0520 05:21:51.276445 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0520 05:21:51.282443 281473412068224 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-3]: I0520 05:21:51.283521 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-3]: I0520 05:21:51.283967 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0520 05:21:51.284422 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-2]: Traceback (most recent call last): [worker-2]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-2]: self.run() [worker-2]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-2]: self._target(*self._args, **self._kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: if self._termination_watcher_fn(): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0520 05:21:51.287994 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0520 05:21:51.288490 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0520 05:21:51.288663 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0520 05:21:51.298595 281473412068224 failure_handling.py:683] Start polling for termination signal. [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-1]: Traceback (most recent call last): [worker-1]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-1]: self.run() [worker-1]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: if self._termination_watcher_fn(): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: I0520 05:21:51.284816 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: self.run() [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: File "/usr/lib/python3.10/threading.py", line 953, in run [worker-1]: I0520 05:21:51.303649 281473412068224 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: self._target(*self._args, **self._kwargs) [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: Instructions for updating: [worker-3]: if self._termination_watcher_fn(): [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: W0520 05:21:51.304084 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: Instructions for updating: [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0520 05:21:51.304250 281473412068224 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:51.394627 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:51.430303 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:51.596721 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:51.675770 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:51.872301 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:51.869923 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:51.878461 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:51.890795 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:51.972616 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:51.990347 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:52.010879 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:52.030740 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:52.150898 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:52.146452 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:52.131208 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:52.160310 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:52.384220 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:52.390295 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:52.402570 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:52.400616 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0520 05:21:52.567753 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5880d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0520 05:21:52.568050 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0520 05:21:52.568754 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c5840d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:52.576774 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:52.577454 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff9c588040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0520 05:21:52.576378 281473412068224 polymorphic_function.py:161] 5 out of the last 5 calls to .wrapped_fn at 0xffff9c588040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:52.577241 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:52.620927 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0520 05:21:52.685485 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0520 05:21:52.692204 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 0 finished [worker-3]: INFO:tensorflow:epoch 0 finished [worker-0]: I0520 05:21:52.692680 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: I0520 05:21:52.685990 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0520 05:21:52.699151 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c588790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0520 05:21:52.699590 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0520 05:21:52.699455 281473412068224 polymorphic_function.py:161] 6 out of the last 6 calls to .wrapped_fn at 0xffff9c584790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0520 05:21:52.699924 281473412068224 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:52.708467 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:52.709217 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:52.709149 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:52.732073 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:52.871360 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:52.872038 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:52.880841 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:52.891477 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:53.106899 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:53.099600 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:53.130443 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:53.157883 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:53.414619 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:53.460435 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:53.510867 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:53.517223 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:53.686315 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:53.700769 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:53.702106 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:53.718411 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:53.876605 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:53.882416 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:53.899461 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:53.923204 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 1 finished [worker-3]: INFO:tensorflow:epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0520 05:21:54.109195 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0520 05:21:54.125619 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: I0520 05:21:54.120049 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: I0520 05:21:54.114781 281473412068224 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:54.131326 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:54.150797 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:54.171722 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:54.191591 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:54.422382 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:54.496913 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:54.520968 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:54.566420 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:54.721081 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:54.745358 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:54.761436 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:54.790783 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:55.005144 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:54.995056 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:55.013181 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:55.042055 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:55.200447 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:55.206475 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:55.264951 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:55.250843 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:55.349316 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:55.346380 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:55.347091 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:55.355282 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0520 05:21:55.432297 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0520 05:21:55.449618 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0520 05:21:55.451148 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I0520 05:21:55.454308 281473412068224 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:55.463685 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:55.471629 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:55.461334 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:55.521590 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:55.585993 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:55.600696 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:55.615680 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:55.631965 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:55.866664 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:55.878195 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:55.907529 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:55.918056 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:55.982338 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:55.988734 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:56.028250 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:56.047737 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:56.147118 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:56.154603 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:56.159899 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:56.152781 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:56.253321 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:56.242325 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:56.342899 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:56.357517 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-3]: I0520 05:21:56.534643 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: I0520 05:21:56.535281 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-1]: I0520 05:21:56.535189 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: I0520 05:21:56.534850 281473412068224 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:56.544176 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:56.544985 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:56.690996 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:56.752268 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:56.845669 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:56.865669 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:56.850526 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:56.871238 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:57.001191 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:57.004201 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:57.010838 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:57.050529 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:57.158336 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:57.151384 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:57.189769 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:57.262256 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:57.465734 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:57.470412 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:57.490355 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:57.537110 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0520 05:21:57.596230 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0520 05:21:57.670979 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0520 05:21:57.689550 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0520 05:21:57.688482 281473412068224 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0520 05:21:57.857105 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0520 05:21:57.857530 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I0520 05:21:57.860000 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0520 05:21:57.861755 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:Training finished. [worker-1]: INFO:tensorflow:Training finished. [worker-2]: I0520 05:21:57.862127 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-1]: I0520 05:21:57.860358 281473412068224 gce_failure_handler_test.py:244] Training finished. [worker-0]: INFO:tensorflow:epoch 4 finished [worker-0]: I0520 05:21:57.869685 281473412068224 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I0520 05:21:57.870080 281473412068224 gce_failure_handler_test.py:244] Training finished. I0520 05:21:58.996320 281473279947648 multi_process_runner.py:646] worker-0 exit code: 0 I0520 05:21:58.996666 281473279947648 multi_process_runner.py:646] worker-1 exit code: 0 I0520 05:21:58.996806 281473279947648 multi_process_runner.py:646] worker-2 exit code: 0 I0520 05:21:58.996931 281473279947648 multi_process_runner.py:646] worker-3 exit code: 0 I0520 05:21:59.004002 281473279947648 multi_process_runner.py:662] Joining log reading threads. I0520 05:21:59.004326 281473279947648 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_7_inputarg_manager_strategyoption_MWMSmultiworker): 18.26s I0520 05:21:59.561021 281473279947648 test_util.py:2464] time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_7_inputarg_manager_strategyoption_MWMSmultiworker): 18.26s [ OK ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_7_inputarg_manager_strategyoption_MWMSmultiworker ====================================================================== FAIL: test_basic_run_test_inputarg_manager_strategyoption_MWMSmultiworker (__main__.GceFailureHandlingTest) GceFailureHandlingTest.test_basic_run_test_inputarg_manager_strategyoption_MWMSmultiworker test_basic_run_test_inputarg_manager_strategyoption_MWMSmultiworker(input_arg='manager', strategy_option='MWMS_multi_worker') ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 329, in test_basic_run mpr.join(timeout=250) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 649, in join self._reraise_if_subprocess_error(process_statuses) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 565, in _reraise_if_subprocess_error six.reraise(*process_status.exc_info) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise raise value File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained return_value = fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 211, in worker_fn self.assertNotEmpty(checkpoint_index) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 972, in assertNotEmpty self.fail('{!r} has length of 0.'.format(container), msg) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 1814, in fail return super(TestCase, self).fail(self._formatMessage(prefix, msg)) File "/usr/lib/python3.10/unittest/case.py", line 675, in fail raise self.failureException(msg) AssertionError: [] has length of 0. ---------------------------------------------------------------------- Ran 7 tests in 111.502s FAILED (failures=1) ================================================================================ ==================== Test output for //tensorflow/python/kernel_tests/nn_ops:conv_ops_test_cpu (shard 3 of 4): 2023-05-20 05:31:22.514186: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. Running tests under Python 3.10.11: /usr/local/bin/python3 [ RUN ] Conv2DTest.testConv2D1x1Padding INFO:tensorflow:Running testConv2D1x1Padding in GRAPH mode. I0520 05:31:31.381007 281472972518272 test_util.py:1494] Running testConv2D1x1Padding in GRAPH mode. WARNING:tensorflow:From /usr/lib/python3.10/contextlib.py:103: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `self.session()` or `self.cached_session()` instead. W0520 05:31:31.381553 281472972518272 deprecation.py:364] From /usr/lib/python3.10/contextlib.py:103: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `self.session()` or `self.cached_session()` instead. 2023-05-20 05:31:31.807529: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:375] MLIR V1 optimization pass is not enabled WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py:154: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. W0520 05:31:31.849613 281472972518272 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py:154: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. 2023-05-20 05:31:31.850361: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D1x1Padding): 0.65s I0520 05:31:32.033970 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2D1x1Padding): 0.65s INFO:tensorflow:Running testConv2D1x1Padding in EAGER mode. I0520 05:31:32.036557 281472972518272 test_util.py:1503] Running testConv2D1x1Padding in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D1x1Padding): 0.2s I0520 05:31:32.234312 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2D1x1Padding): 0.2s [ OK ] Conv2DTest.testConv2D1x1Padding [ RUN ] Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput INFO:tensorflow:Running testConv2D2x2Depth1Padding0x0BackpropInput in GRAPH mode. I0520 05:31:32.236501 281472972518272 test_util.py:1494] Running testConv2D2x2Depth1Padding0x0BackpropInput in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput): 0.13s I0520 05:31:32.368601 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput): 0.13s INFO:tensorflow:Running testConv2D2x2Depth1Padding0x0BackpropInput in EAGER mode. I0520 05:31:32.370615 281472972518272 test_util.py:1503] Running testConv2D2x2Depth1Padding0x0BackpropInput in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput): 0.07s I0520 05:31:32.442962 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput): 0.07s [ OK ] Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput [ RUN ] Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput INFO:tensorflow:Running testConv2D2x2Depth1Padding2x2BackpropInput in GRAPH mode. I0520 05:31:32.445111 281472972518272 test_util.py:1494] Running testConv2D2x2Depth1Padding2x2BackpropInput in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput): 0.06s I0520 05:31:32.508837 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput): 0.06s INFO:tensorflow:Running testConv2D2x2Depth1Padding2x2BackpropInput in EAGER mode. I0520 05:31:32.510509 281472972518272 test_util.py:1503] Running testConv2D2x2Depth1Padding2x2BackpropInput in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput): 0.06s I0520 05:31:32.575283 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput): 0.06s [ OK ] Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput [ RUN ] Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput INFO:tensorflow:Running testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput in GRAPH mode. I0520 05:31:32.577249 281472972518272 test_util.py:1494] Running testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput): 0.06s I0520 05:31:32.637093 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput): 0.06s INFO:tensorflow:Running testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput in EAGER mode. I0520 05:31:32.638748 281472972518272 test_util.py:1503] Running testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput): 0.03s I0520 05:31:32.672930 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput): 0.03s [ OK ] Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput [ RUN ] Conv2DTest.testConv2D2x2Depth1ValidBackpropInputDilation1x2 INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1ValidBackpropInputDilation1x2): 0.12s I0520 05:31:32.794762 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1ValidBackpropInputDilation1x2): 0.12s [ OK ] Conv2DTest.testConv2D2x2Depth1ValidBackpropInputDilation1x2 [ RUN ] Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2 INFO:tensorflow:Running testConv2D2x2Depth3ValidBackpropFilterStride1x2 in GRAPH mode. I0520 05:31:32.796831 281472972518272 test_util.py:1494] Running testConv2D2x2Depth3ValidBackpropFilterStride1x2 in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2): 0.06s I0520 05:31:32.853728 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2): 0.06s INFO:tensorflow:Running testConv2D2x2Depth3ValidBackpropFilterStride1x2 in EAGER mode. I0520 05:31:32.855338 281472972518272 test_util.py:1503] Running testConv2D2x2Depth3ValidBackpropFilterStride1x2 in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2): 0.03s I0520 05:31:32.885539 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2): 0.03s [ OK ] Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2 [ RUN ] Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2 INFO:tensorflow:Running testConv2D2x2Depth3ValidBackpropInputStride1x2 in GRAPH mode. I0520 05:31:32.887146 281472972518272 test_util.py:1494] Running testConv2D2x2Depth3ValidBackpropInputStride1x2 in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2): 0.03s I0520 05:31:32.916544 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2): 0.03s INFO:tensorflow:Running testConv2D2x2Depth3ValidBackpropInputStride1x2 in EAGER mode. I0520 05:31:32.918066 281472972518272 test_util.py:1503] Running testConv2D2x2Depth3ValidBackpropInputStride1x2 in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2): 0.03s I0520 05:31:32.943199 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2): 0.03s [ OK ] Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2 [ RUN ] Conv2DTest.testConv2D2x2FilterStride1x2 INFO:tensorflow:Running testConv2D2x2FilterStride1x2 in GRAPH mode. I0520 05:31:32.944733 281472972518272 test_util.py:1494] Running testConv2D2x2FilterStride1x2 in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2FilterStride1x2): 0.1s I0520 05:31:33.042449 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2FilterStride1x2): 0.1s INFO:tensorflow:Running testConv2D2x2FilterStride1x2 in EAGER mode. I0520 05:31:33.044241 281472972518272 test_util.py:1503] Running testConv2D2x2FilterStride1x2 in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2FilterStride1x2): 0.03s I0520 05:31:33.073589 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2FilterStride1x2): 0.03s [ OK ] Conv2DTest.testConv2D2x2FilterStride1x2 [ RUN ] Conv2DTest.testConv2DBackpropFilterWithEmptyInput INFO:tensorflow:Running testConv2DBackpropFilterWithEmptyInput in GRAPH mode. I0520 05:31:33.075236 281472972518272 test_util.py:1494] Running testConv2DBackpropFilterWithEmptyInput in GRAPH mode. 2023-05-20 05:31:33.089738: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2DBackpropFilter" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 1 } } } } inputs { dtype: DT_INT32 shape { dim { size: 4 } } value { dtype: DT_INT32 tensor_shape { dim { size: 4 } } tensor_content: "\002\000\000\000\002\000\000\000\001\000\000\000\001\000\000\000" } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 1 } dim { size: 2 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 1 } dim { size: 2 } dim { size: 1 } } } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: 2 } dim { size: 2 } dim { size: 1 } dim { size: 1 } } } 2023-05-20 05:31:33.119841: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2DBackpropFilter" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 1 } } } } inputs { dtype: DT_INT32 shape { dim { size: 4 } } value { dtype: DT_INT32 tensor_shape { dim { size: 4 } } tensor_content: "\002\000\000\000\002\000\000\000\001\000\000\000\001\000\000\000" } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 1 } dim { size: 2 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 1 } dim { size: 2 } dim { size: 1 } } } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: 2 } dim { size: 2 } dim { size: 1 } dim { size: 1 } } } INFO:tensorflow:time(__main__.Conv2DTest.testConv2DBackpropFilterWithEmptyInput): 0.08s I0520 05:31:33.150747 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2DBackpropFilterWithEmptyInput): 0.08s INFO:tensorflow:Running testConv2DBackpropFilterWithEmptyInput in EAGER mode. I0520 05:31:33.152423 281472972518272 test_util.py:1503] Running testConv2DBackpropFilterWithEmptyInput in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DBackpropFilterWithEmptyInput): 0.01s I0520 05:31:33.167162 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2DBackpropFilterWithEmptyInput): 0.01s [ OK ] Conv2DTest.testConv2DBackpropFilterWithEmptyInput [ RUN ] Conv2DTest.testConv2DEmptyBackpropFilter INFO:tensorflow:Running testConv2DEmptyBackpropFilter in GRAPH mode. I0520 05:31:33.168705 281472972518272 test_util.py:1494] Running testConv2DEmptyBackpropFilter in GRAPH mode. 2023-05-20 05:31:33.181350: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2DBackpropFilter" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 2 } dim { size: 3 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 2 } dim { size: 3 } dim { size: 1 } } tensor_content: "\000\000\200?\000\000\000@\000\000@@\000\000\200@\000\000\240@\000\000\300@" } } inputs { dtype: DT_INT32 shape { dim { size: 4 } } value { dtype: DT_INT32 tensor_shape { dim { size: 4 } } tensor_content: "\002\000\000\000\002\000\000\000\001\000\000\000\000\000\000\000" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1 } dim { size: 2 } dim { } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 1 } dim { size: 2 } dim { } } } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: 2 } dim { size: 2 } dim { size: 1 } dim { } } } 2023-05-20 05:31:33.203151: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2DBackpropFilter" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 2 } dim { size: 3 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 2 } dim { size: 3 } dim { size: 1 } } tensor_content: "\000\000\200?\000\000\000@\000\000@@\000\000\200@\000\000\240@\000\000\300@" } } inputs { dtype: DT_INT32 shape { dim { size: 4 } } value { dtype: DT_INT32 tensor_shape { dim { size: 4 } } tensor_content: "\002\000\000\000\002\000\000\000\001\000\000\000\000\000\000\000" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1 } dim { size: 2 } dim { } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 1 } dim { size: 2 } dim { } } } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: 2 } dim { size: 2 } dim { size: 1 } dim { } } } INFO:tensorflow:time(__main__.Conv2DTest.testConv2DEmptyBackpropFilter): 0.05s I0520 05:31:33.213700 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2DEmptyBackpropFilter): 0.05s INFO:tensorflow:Running testConv2DEmptyBackpropFilter in EAGER mode. I0520 05:31:33.215273 281472972518272 test_util.py:1503] Running testConv2DEmptyBackpropFilter in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DEmptyBackpropFilter): 0.01s I0520 05:31:33.229310 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2DEmptyBackpropFilter): 0.01s [ OK ] Conv2DTest.testConv2DEmptyBackpropFilter [ RUN ] Conv2DTest.testConv2DEmptyDilation INFO:tensorflow:Running testConv2DEmptyDilation in GRAPH mode. I0520 05:31:33.230793 281472972518272 test_util.py:1494] Running testConv2DEmptyDilation in GRAPH mode. 2023-05-20 05:31:33.282031: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2D" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 1 } dim { size: 3 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } tensor_content: "\000\000\200?\000\000\000@\000\000@@\000\000\200@\000\000\240@\000\000\300@\000\000\340@\000\000\000A\000\000\020A" } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { } dim { size: 1 } dim { size: 3 } dim { size: 3 } } } 2023-05-20 05:31:33.299187: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2D" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 2 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } tensor_content: "\000\000\200?\000\000\000@\000\000@@\000\000\200@\000\000\240@\000\000\300@\000\000\340@\000\000\000A\000\000\020A" } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } } 2023-05-20 05:31:33.299569: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2D" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 2 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } tensor_content: "\000\000\200?\000\000\000@\000\000@@\000\000\200@\000\000\240@\000\000\300@\000\000\340@\000\000\000A\000\000\020A" } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } } INFO:tensorflow:time(__main__.Conv2DTest.testConv2DEmptyDilation): 0.08s I0520 05:31:33.312687 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2DEmptyDilation): 0.08s INFO:tensorflow:Running testConv2DEmptyDilation in EAGER mode. I0520 05:31:33.314482 281472972518272 test_util.py:1503] Running testConv2DEmptyDilation in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DEmptyDilation): 0.03s I0520 05:31:33.344315 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2DEmptyDilation): 0.03s [ OK ] Conv2DTest.testConv2DEmptyDilation [ RUN ] Conv2DTest.testConv2DGroupConvFwd INFO:tensorflow:Running testConv2DGroupConvFwd in GRAPH mode. I0520 05:31:33.345985 281472972518272 test_util.py:1494] Running testConv2DGroupConvFwd in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DGroupConvFwd): 1.01s I0520 05:31:34.360160 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2DGroupConvFwd): 1.01s [ FAILED ] Conv2DTest.testConv2DGroupConvFwd [ RUN ] Conv2DTest.testConv2DKernelSizeMatchesInputSizeBackpropFilterDilation2x2 INFO:tensorflow:time(__main__.Conv2DTest.testConv2DKernelSizeMatchesInputSizeBackpropFilterDilation2x2): 0.09s I0520 05:31:34.460912 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2DKernelSizeMatchesInputSizeBackpropFilterDilation2x2): 0.09s [ OK ] Conv2DTest.testConv2DKernelSizeMatchesInputSizeBackpropFilterDilation2x2 [ RUN ] Conv2DTest.testConv2DKernelSmallerThanStrideSame INFO:tensorflow:Running testConv2DKernelSmallerThanStrideSame in GRAPH mode. I0520 05:31:34.463042 281472972518272 test_util.py:1494] Running testConv2DKernelSmallerThanStrideSame in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DKernelSmallerThanStrideSame): 0.24s I0520 05:31:34.703502 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2DKernelSmallerThanStrideSame): 0.24s INFO:tensorflow:Running testConv2DKernelSmallerThanStrideSame in EAGER mode. I0520 05:31:34.706356 281472972518272 test_util.py:1503] Running testConv2DKernelSmallerThanStrideSame in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DKernelSmallerThanStrideSame): 0.07s I0520 05:31:34.773218 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2DKernelSmallerThanStrideSame): 0.07s [ OK ] Conv2DTest.testConv2DKernelSmallerThanStrideSame [ RUN ] Conv2DTest.testConv2DOnlyTopRightPadding INFO:tensorflow:Running testConv2DOnlyTopRightPadding in GRAPH mode. I0520 05:31:34.775013 281472972518272 test_util.py:1494] Running testConv2DOnlyTopRightPadding in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DOnlyTopRightPadding): 0.21s I0520 05:31:34.989743 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2DOnlyTopRightPadding): 0.21s INFO:tensorflow:Running testConv2DOnlyTopRightPadding in EAGER mode. I0520 05:31:34.992312 281472972518272 test_util.py:1503] Running testConv2DOnlyTopRightPadding in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DOnlyTopRightPadding): 0.09s I0520 05:31:35.081671 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConv2DOnlyTopRightPadding): 0.09s [ OK ] Conv2DTest.testConv2DOnlyTopRightPadding [ RUN ] Conv2DTest.testConvolutionClass2DExpandedBatch INFO:tensorflow:Running testConvolutionClass2DExpandedBatch in GRAPH mode. I0520 05:31:35.084108 281472972518272 test_util.py:1494] Running testConvolutionClass2DExpandedBatch in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConvolutionClass2DExpandedBatch): 0.07s I0520 05:31:35.155008 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConvolutionClass2DExpandedBatch): 0.07s INFO:tensorflow:Running testConvolutionClass2DExpandedBatch in EAGER mode. I0520 05:31:35.156787 281472972518272 test_util.py:1503] Running testConvolutionClass2DExpandedBatch in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConvolutionClass2DExpandedBatch): 0.04s I0520 05:31:35.194905 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testConvolutionClass2DExpandedBatch): 0.04s [ OK ] Conv2DTest.testConvolutionClass2DExpandedBatch [ RUN ] Conv2DTest.testFilterGradient1x1PaddingStrideOne WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py:1953: compute_gradient (from tensorflow.python.ops.gradient_checker) is deprecated and will be removed in a future version. Instructions for updating: Use tf.test.compute_gradient in 2.0, which has better support for functions. Note that the two versions have different usage, so code change is needed. W0520 05:31:35.204527 281472972518272 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py:1953: compute_gradient (from tensorflow.python.ops.gradient_checker) is deprecated and will be removed in a future version. Instructions for updating: Use tf.test.compute_gradient in 2.0, which has better support for functions. Note that the two versions have different usage, so code change is needed. INFO:tensorflow:time(__main__.Conv2DTest.testFilterGradient1x1PaddingStrideOne): 0.73s I0520 05:31:35.922782 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testFilterGradient1x1PaddingStrideOne): 0.73s [ OK ] Conv2DTest.testFilterGradient1x1PaddingStrideOne [ RUN ] Conv2DTest.testFilterGradientGroupConv INFO:tensorflow:time(__main__.Conv2DTest.testFilterGradientGroupConv): 0.0s I0520 05:31:35.925316 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testFilterGradientGroupConv): 0.0s [ SKIPPED ] Conv2DTest.testFilterGradientGroupConv [ RUN ] Conv2DTest.testFilterGradientSamePaddingStrideThree INFO:tensorflow:time(__main__.Conv2DTest.testFilterGradientSamePaddingStrideThree): 1.92s I0520 05:31:37.847131 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testFilterGradientSamePaddingStrideThree): 1.92s [ OK ] Conv2DTest.testFilterGradientSamePaddingStrideThree [ RUN ] Conv2DTest.testFilterGradientValidPaddingStrideTwo INFO:tensorflow:time(__main__.Conv2DTest.testFilterGradientValidPaddingStrideTwo): 1.34s I0520 05:31:39.193435 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testFilterGradientValidPaddingStrideTwo): 1.34s [ OK ] Conv2DTest.testFilterGradientValidPaddingStrideTwo [ RUN ] Conv2DTest.testInceptionBackFilter_11 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:39.195340 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], [1, 1], 'SAME') I0520 05:31:39.196578 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_11): 0.06s I0520 05:31:39.256961 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_11): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:39.258600 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], [1, 1], 'SAME') I0520 05:31:39.258918 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_11): 0.03s I0520 05:31:39.290899 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_11): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_11 [ RUN ] Conv2DTest.testInceptionBackFilter_15 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:39.292309 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], [2, 2], 'VALID') I0520 05:31:39.293648 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], [2, 2], 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_15): 0.07s I0520 05:31:39.361092 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_15): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:39.362656 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], [2, 2], 'VALID') I0520 05:31:39.362950 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], [2, 2], 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_15): 0.03s I0520 05:31:39.388170 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_15): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_15 [ RUN ] Conv2DTest.testInceptionBackFilter_19 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:39.389558 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], [1, 1], 'SAME') I0520 05:31:39.390837 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_19): 0.07s I0520 05:31:39.460158 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_19): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:39.461779 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], [1, 1], 'SAME') I0520 05:31:39.462095 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_19): 0.03s I0520 05:31:39.492775 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_19): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_19 [ RUN ] Conv2DTest.testInceptionBackFilter_22 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:39.494073 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], [1, 1], 'SAME') I0520 05:31:39.495312 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_22): 0.06s I0520 05:31:39.554494 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_22): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:39.556015 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], [1, 1], 'SAME') I0520 05:31:39.556366 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_22): 0.03s I0520 05:31:39.582873 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_22): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_22 [ RUN ] Conv2DTest.testInceptionBackFilter_26 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:39.584326 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], [1, 1], 'SAME') I0520 05:31:39.585712 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_26): 0.08s I0520 05:31:39.664031 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_26): 0.08s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:39.665544 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], [1, 1], 'SAME') I0520 05:31:39.665832 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_26): 0.03s I0520 05:31:39.696790 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_26): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_26 [ RUN ] Conv2DTest.testInceptionBackFilter_3 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:39.698090 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], [1, 1], 'SAME') I0520 05:31:39.699324 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_3): 0.06s I0520 05:31:39.758846 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_3): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:39.760371 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], [1, 1], 'SAME') I0520 05:31:39.760642 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_3): 0.03s I0520 05:31:39.786868 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_3): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_3 [ RUN ] Conv2DTest.testInceptionBackFilter_33 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:39.788258 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], [1, 1], 'SAME') I0520 05:31:39.789627 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_33): 0.06s I0520 05:31:39.846225 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_33): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:39.847750 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], [1, 1], 'SAME') I0520 05:31:39.848073 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_33): 0.03s I0520 05:31:39.875186 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_33): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_33 [ RUN ] Conv2DTest.testInceptionBackFilter_37 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:39.876561 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], [1, 1], 'SAME') I0520 05:31:39.877890 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_37): 0.07s I0520 05:31:39.945124 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_37): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:39.946660 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], [1, 1], 'SAME') I0520 05:31:39.946930 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_37): 0.03s I0520 05:31:39.980751 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_37): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_37 [ RUN ] Conv2DTest.testInceptionBackFilter_40 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:39.982069 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], [1, 1], 'SAME') I0520 05:31:39.983260 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_40): 0.09s I0520 05:31:40.076557 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_40): 0.09s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:40.077999 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], [1, 1], 'SAME') I0520 05:31:40.078271 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_40): 0.03s I0520 05:31:40.106183 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_40): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_40 [ RUN ] Conv2DTest.testInceptionBackFilter_44 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:40.107456 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], [1, 1], 'SAME') I0520 05:31:40.108599 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_44): 0.08s I0520 05:31:40.188767 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_44): 0.08s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:40.190233 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], [1, 1], 'SAME') I0520 05:31:40.190486 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_44): 0.03s I0520 05:31:40.220959 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_44): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_44 [ RUN ] Conv2DTest.testInceptionBackFilter_48 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:40.222286 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], [1, 1], 'SAME') I0520 05:31:40.223476 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_48): 0.11s I0520 05:31:40.327397 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_48): 0.11s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:40.328904 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], [1, 1], 'SAME') I0520 05:31:40.329164 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_48): 0.07s I0520 05:31:40.399425 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_48): 0.07s [ OK ] Conv2DTest.testInceptionBackFilter_48 [ RUN ] Conv2DTest.testInceptionBackFilter_51 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:40.400815 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], [1, 1], 'SAME') I0520 05:31:40.402036 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_51): 0.07s I0520 05:31:40.469422 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_51): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:40.470939 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], [1, 1], 'SAME') I0520 05:31:40.471322 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_51): 0.02s I0520 05:31:40.495023 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_51): 0.02s [ OK ] Conv2DTest.testInceptionBackFilter_51 [ RUN ] Conv2DTest.testInceptionBackFilter_6 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:40.496464 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], [1, 1], 'SAME') I0520 05:31:40.497758 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_6): 0.06s I0520 05:31:40.559679 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_6): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:40.561197 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], [1, 1], 'SAME') I0520 05:31:40.561513 281472972518272 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_6): 0.04s I0520 05:31:40.597763 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_6): 0.04s [ OK ] Conv2DTest.testInceptionBackFilter_6 [ RUN ] Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:40.599137 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Skipping InceptionBackFilter ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], [1, 1], 'SAME') I0520 05:31:40.600565 281472972518272 conv_ops_test.py:3254] Skipping InceptionBackFilter ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused): 0.0s I0520 05:31:40.600947 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused): 0.0s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:40.601765 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Skipping InceptionBackFilter ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], [1, 1], 'SAME') I0520 05:31:40.602086 281472972518272 conv_ops_test.py:3254] Skipping InceptionBackFilter ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused): 0.0s I0520 05:31:40.602449 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused): 0.0s [ OK ] Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused [ RUN ] Conv2DTest.testInceptionBackInput_11 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:40.603252 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], 1, 'SAME') I0520 05:31:40.604265 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_11): 0.05s I0520 05:31:40.655754 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_11): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:40.657128 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], 1, 'SAME') I0520 05:31:40.657424 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_11): 0.03s I0520 05:31:40.689285 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_11): 0.03s [ OK ] Conv2DTest.testInceptionBackInput_11 [ RUN ] Conv2DTest.testInceptionBackInput_15 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:40.690602 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], 2, 'VALID') I0520 05:31:40.691783 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], 2, 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_15): 0.07s I0520 05:31:40.756381 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_15): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:40.757722 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], 2, 'VALID') I0520 05:31:40.757969 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], 2, 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_15): 0.02s I0520 05:31:40.778178 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_15): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_15 [ RUN ] Conv2DTest.testInceptionBackInput_19 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:40.779479 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], 1, 'SAME') I0520 05:31:40.780677 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_19): 0.05s I0520 05:31:40.834481 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_19): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:40.835931 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], 1, 'SAME') I0520 05:31:40.836235 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_19): 0.02s I0520 05:31:40.853277 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_19): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_19 [ RUN ] Conv2DTest.testInceptionBackInput_22 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:40.854500 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], 1, 'SAME') I0520 05:31:40.855602 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_22): 0.05s I0520 05:31:40.907055 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_22): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:40.908491 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], 1, 'SAME') I0520 05:31:40.908730 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_22): 0.02s I0520 05:31:40.927126 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_22): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_22 [ RUN ] Conv2DTest.testInceptionBackInput_26 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:40.928282 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], 1, 'SAME') I0520 05:31:40.929291 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_26): 0.05s I0520 05:31:40.979440 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_26): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:40.980657 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], 1, 'SAME') I0520 05:31:40.980850 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_26): 0.02s I0520 05:31:40.999599 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_26): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_26 [ RUN ] Conv2DTest.testInceptionBackInput_3 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:41.000683 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], 1, 'SAME') I0520 05:31:41.001612 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_3): 0.06s I0520 05:31:41.060444 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_3): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:41.061588 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], 1, 'SAME') I0520 05:31:41.061786 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_3): 0.04s I0520 05:31:41.104198 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_3): 0.04s [ OK ] Conv2DTest.testInceptionBackInput_3 [ RUN ] Conv2DTest.testInceptionBackInput_33 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:41.105286 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], 1, 'SAME') I0520 05:31:41.106296 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_33): 0.07s I0520 05:31:41.172541 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_33): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:41.173689 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], 1, 'SAME') I0520 05:31:41.173877 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_33): 0.02s I0520 05:31:41.190738 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_33): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_33 [ RUN ] Conv2DTest.testInceptionBackInput_37 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:41.191888 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], 1, 'SAME') I0520 05:31:41.192880 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_37): 0.13s I0520 05:31:41.317514 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_37): 0.13s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:41.318800 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], 1, 'SAME') I0520 05:31:41.318990 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_37): 0.02s I0520 05:31:41.340826 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_37): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_37 [ RUN ] Conv2DTest.testInceptionBackInput_40 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:41.341927 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], 1, 'SAME') I0520 05:31:41.342848 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_40): 0.06s I0520 05:31:41.398864 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_40): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:41.400086 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], 1, 'SAME') I0520 05:31:41.400276 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_40): 0.02s I0520 05:31:41.419665 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_40): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_40 [ RUN ] Conv2DTest.testInceptionBackInput_44 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:41.420790 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], 1, 'SAME') I0520 05:31:41.421751 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_44): 0.07s I0520 05:31:41.495416 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_44): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:41.496730 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], 1, 'SAME') I0520 05:31:41.496955 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_44): 0.02s I0520 05:31:41.516605 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_44): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_44 [ RUN ] Conv2DTest.testInceptionBackInput_48 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:41.517743 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], 1, 'SAME') I0520 05:31:41.518739 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_48): 0.05s I0520 05:31:41.569923 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_48): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:41.571098 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], 1, 'SAME') I0520 05:31:41.571286 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_48): 0.1s I0520 05:31:41.672043 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_48): 0.1s [ OK ] Conv2DTest.testInceptionBackInput_48 [ RUN ] Conv2DTest.testInceptionBackInput_51 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:41.673106 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], 1, 'SAME') I0520 05:31:41.673998 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_51): 0.05s I0520 05:31:41.718449 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_51): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:41.719562 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], 1, 'SAME') I0520 05:31:41.719746 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_51): 0.02s I0520 05:31:41.735797 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_51): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_51 [ RUN ] Conv2DTest.testInceptionBackInput_6 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:41.736838 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], 1, 'SAME') I0520 05:31:41.737649 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_6): 0.05s I0520 05:31:41.785497 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_6): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:41.786754 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], 1, 'SAME') I0520 05:31:41.786953 281472972518272 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_6): 0.02s I0520 05:31:41.803176 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_6): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_6 [ RUN ] Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:41.804271 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Skipping InceptionBackInput ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], 1, 'SAME') I0520 05:31:41.805244 281472972518272 conv_ops_test.py:3238] Skipping InceptionBackInput ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused): 0.0s I0520 05:31:41.805498 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused): 0.0s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:41.806071 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Skipping InceptionBackInput ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], 1, 'SAME') I0520 05:31:41.806331 281472972518272 conv_ops_test.py:3238] Skipping InceptionBackInput ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused): 0.0s I0520 05:31:41.806606 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused): 0.0s [ OK ] Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_11 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:41.807244 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') I0520 05:31:41.807978 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_11): 0.08s I0520 05:31:41.892180 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_11): 0.08s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:41.893503 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') I0520 05:31:41.893704 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_11): 0.04s I0520 05:31:41.932943 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_11): 0.04s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_11 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_15 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:41.934370 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_15): 0.0s I0520 05:31:41.935604 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_15): 0.0s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:41.936356 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_15): 0.0s I0520 05:31:41.936685 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_15): 0.0s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_15 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_19 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:41.937355 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') I0520 05:31:41.938228 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_19): 0.1s I0520 05:31:42.036341 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_19): 0.1s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:42.038070 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') I0520 05:31:42.038349 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_19): 0.06s I0520 05:31:42.096594 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_19): 0.06s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_19 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_22 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:42.098021 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') I0520 05:31:42.099116 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_22): 0.15s I0520 05:31:42.243829 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_22): 0.15s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:42.245413 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') I0520 05:31:42.245660 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_22): 0.06s I0520 05:31:42.302912 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_22): 0.06s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_22 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_26 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:42.304317 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') I0520 05:31:42.305441 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_26): 0.13s I0520 05:31:42.431474 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_26): 0.13s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:42.432977 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') I0520 05:31:42.433194 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_26): 0.05s I0520 05:31:42.484038 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_26): 0.05s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_26 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_3 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:42.485457 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') I0520 05:31:42.486728 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_3): 0.13s I0520 05:31:42.611797 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_3): 0.13s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:42.613346 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') I0520 05:31:42.613532 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_3): 0.12s I0520 05:31:42.728483 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_3): 0.12s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_3 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_33 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:42.729833 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') I0520 05:31:42.730806 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_33): 0.11s I0520 05:31:42.842092 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_33): 0.11s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:42.843548 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') I0520 05:31:42.843734 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_33): 0.23s I0520 05:31:43.069600 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_33): 0.23s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_33 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_37 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:43.070914 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') I0520 05:31:43.071958 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_37): 0.09s I0520 05:31:43.161541 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_37): 0.09s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:43.162962 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') I0520 05:31:43.163177 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_37): 0.04s I0520 05:31:43.206969 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_37): 0.04s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_37 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_40 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:43.208335 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') I0520 05:31:43.209790 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_40): 0.11s I0520 05:31:43.317620 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_40): 0.11s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:43.319098 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') I0520 05:31:43.319294 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_40): 0.04s I0520 05:31:43.355423 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_40): 0.04s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_40 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_44 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:43.356840 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') I0520 05:31:43.357888 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_44): 0.29s I0520 05:31:43.648847 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_44): 0.29s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:43.650235 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') I0520 05:31:43.650434 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_44): 0.04s I0520 05:31:43.689085 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_44): 0.04s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_44 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_48 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:43.690419 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') I0520 05:31:43.691446 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_48): 0.16s I0520 05:31:43.848686 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_48): 0.16s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:43.850133 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') I0520 05:31:43.850355 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_48): 0.04s I0520 05:31:43.894662 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_48): 0.04s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_48 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_51 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:43.896048 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') I0520 05:31:43.897082 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_51): 0.16s I0520 05:31:44.054276 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_51): 0.16s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:44.055678 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') I0520 05:31:44.055855 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_51): 0.18s I0520 05:31:44.235392 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_51): 0.18s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_51 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_6 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:44.236718 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') I0520 05:31:44.237725 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_6): 0.16s I0520 05:31:44.398112 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_6): 0.16s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:44.399489 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') I0520 05:31:44.399699 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_6): 0.04s I0520 05:31:44.443936 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_6): 0.04s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_6 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:44.445307 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') I0520 05:31:44.446265 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused): 1.45s I0520 05:31:45.895605 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused): 1.45s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:45.897043 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') I0520 05:31:45.897261 281472972518272 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused): 2.02s I0520 05:31:47.913938 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused): 2.02s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused [ RUN ] Conv2DTest.testInceptionFwd_11 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:47.915432 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') I0520 05:31:47.916696 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_11): 0.08s I0520 05:31:47.997459 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_11): 0.08s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:47.998662 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') I0520 05:31:47.998896 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_11): 0.01s I0520 05:31:48.010748 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_11): 0.01s [ OK ] Conv2DTest.testInceptionFwd_11 [ RUN ] Conv2DTest.testInceptionFwd_15 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:48.011842 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [3, 3, 12, 32], 2, 'VALID') I0520 05:31:48.012844 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [3, 3, 12, 32], 2, 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_15): 0.04s I0520 05:31:48.051956 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_15): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:48.053216 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [3, 3, 12, 32], 2, 'VALID') I0520 05:31:48.053466 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [3, 3, 12, 32], 2, 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_15): 0.02s I0520 05:31:48.068667 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_15): 0.02s [ OK ] Conv2DTest.testInceptionFwd_15 [ RUN ] Conv2DTest.testInceptionFwd_19 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:48.069813 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') I0520 05:31:48.070871 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_19): 0.04s I0520 05:31:48.106231 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_19): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:48.107367 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') I0520 05:31:48.107588 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_19): 0.01s I0520 05:31:48.122340 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_19): 0.01s [ OK ] Conv2DTest.testInceptionFwd_19 [ RUN ] Conv2DTest.testInceptionFwd_22 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:48.123477 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') I0520 05:31:48.124488 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_22): 0.03s I0520 05:31:48.157571 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_22): 0.03s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:48.158645 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') I0520 05:31:48.158853 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_22): 0.01s I0520 05:31:48.169561 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_22): 0.01s [ OK ] Conv2DTest.testInceptionFwd_22 [ RUN ] Conv2DTest.testInceptionFwd_26 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:48.170638 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') I0520 05:31:48.171577 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_26): 0.05s I0520 05:31:48.219112 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_26): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:48.220273 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') I0520 05:31:48.220516 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_26): 0.06s I0520 05:31:48.278479 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_26): 0.06s [ OK ] Conv2DTest.testInceptionFwd_26 [ RUN ] Conv2DTest.testInceptionFwd_3 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:48.282111 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') I0520 05:31:48.283205 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_3): 0.04s I0520 05:31:48.319865 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_3): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:48.321020 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') I0520 05:31:48.321233 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_3): 0.01s I0520 05:31:48.333898 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_3): 0.01s [ OK ] Conv2DTest.testInceptionFwd_3 [ RUN ] Conv2DTest.testInceptionFwd_33 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:48.335099 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') I0520 05:31:48.336108 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_33): 0.03s I0520 05:31:48.366324 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_33): 0.03s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:48.367464 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') I0520 05:31:48.367705 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_33): 0.02s I0520 05:31:48.389415 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_33): 0.02s [ OK ] Conv2DTest.testInceptionFwd_33 [ RUN ] Conv2DTest.testInceptionFwd_37 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:48.390542 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') I0520 05:31:48.391506 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_37): 0.04s I0520 05:31:48.428242 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_37): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:48.429379 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') I0520 05:31:48.429603 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_37): 0.01s I0520 05:31:48.442559 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_37): 0.01s [ OK ] Conv2DTest.testInceptionFwd_37 [ RUN ] Conv2DTest.testInceptionFwd_40 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:48.443723 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') I0520 05:31:48.444798 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_40): 0.04s I0520 05:31:48.485619 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_40): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:48.486835 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') I0520 05:31:48.487080 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_40): 0.01s I0520 05:31:48.500860 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_40): 0.01s [ OK ] Conv2DTest.testInceptionFwd_40 [ RUN ] Conv2DTest.testInceptionFwd_44 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:48.501926 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') I0520 05:31:48.502867 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_44): 0.04s I0520 05:31:48.545633 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_44): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:48.546832 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') I0520 05:31:48.547090 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_44): 0.02s I0520 05:31:48.562719 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_44): 0.02s [ OK ] Conv2DTest.testInceptionFwd_44 [ RUN ] Conv2DTest.testInceptionFwd_48 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:48.563855 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') I0520 05:31:48.564910 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_48): 0.04s I0520 05:31:48.607308 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_48): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:48.608482 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') I0520 05:31:48.608733 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_48): 0.01s I0520 05:31:48.622469 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_48): 0.01s [ OK ] Conv2DTest.testInceptionFwd_48 [ RUN ] Conv2DTest.testInceptionFwd_51 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:48.623521 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') I0520 05:31:48.624428 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_51): 0.04s I0520 05:31:48.664410 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_51): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:48.665577 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') I0520 05:31:48.665800 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_51): 0.02s I0520 05:31:48.685176 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_51): 0.02s [ OK ] Conv2DTest.testInceptionFwd_51 [ RUN ] Conv2DTest.testInceptionFwd_6 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:48.686354 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') I0520 05:31:48.687285 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_6): 0.04s I0520 05:31:48.723147 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_6): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:48.724340 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') I0520 05:31:48.724575 281472972518272 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_6): 0.01s I0520 05:31:48.737475 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_6): 0.01s [ OK ] Conv2DTest.testInceptionFwd_6 [ RUN ] Conv2DTest.testInceptionFwd_No_Winograd_Nonfused INFO:tensorflow:Running Test in GRAPH mode. I0520 05:31:48.738591 281472972518272 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Skipping InceptionFwd ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') I0520 05:31:48.739601 281472972518272 conv_ops_test.py:3205] Skipping InceptionFwd ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_No_Winograd_Nonfused): 0.0s I0520 05:31:48.739896 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_No_Winograd_Nonfused): 0.0s INFO:tensorflow:Running Test in EAGER mode. I0520 05:31:48.740526 281472972518272 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Skipping InceptionFwd ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') I0520 05:31:48.740767 281472972518272 conv_ops_test.py:3205] Skipping InceptionFwd ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_No_Winograd_Nonfused): 0.0s I0520 05:31:48.741048 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_No_Winograd_Nonfused): 0.0s [ OK ] Conv2DTest.testInceptionFwd_No_Winograd_Nonfused [ RUN ] Conv2DTest.testInputGradient1x1PaddingStrideTwo INFO:tensorflow:time(__main__.Conv2DTest.testInputGradient1x1PaddingStrideTwo): 0.38s I0520 05:31:49.120054 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInputGradient1x1PaddingStrideTwo): 0.38s [ OK ] Conv2DTest.testInputGradient1x1PaddingStrideTwo [ RUN ] Conv2DTest.testInputGradientKernelSizeMatchesInputSize INFO:tensorflow:time(__main__.Conv2DTest.testInputGradientKernelSizeMatchesInputSize): 0.28s I0520 05:31:49.398446 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInputGradientKernelSizeMatchesInputSize): 0.28s [ OK ] Conv2DTest.testInputGradientKernelSizeMatchesInputSize [ RUN ] Conv2DTest.testInputGradientValidPaddingStrideOne INFO:tensorflow:time(__main__.Conv2DTest.testInputGradientValidPaddingStrideOne): 0.5s I0520 05:31:49.899883 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testInputGradientValidPaddingStrideOne): 0.5s [ OK ] Conv2DTest.testInputGradientValidPaddingStrideOne [ RUN ] Conv2DTest.testShapeFunctionEdgeCases INFO:tensorflow:time(__main__.Conv2DTest.testShapeFunctionEdgeCases): 0.03s I0520 05:31:49.931805 281472972518272 test_util.py:2464] time(__main__.Conv2DTest.testShapeFunctionEdgeCases): 0.03s [ OK ] Conv2DTest.testShapeFunctionEdgeCases [ RUN ] DeepConv2DTest.test_session [ SKIPPED ] DeepConv2DTest.test_session [ RUN ] FusedConv2DTest.testAddWithRefCountTwoAndNoDependence INFO:tensorflow:Running testAddWithRefCountTwoAndNoDependence in GRAPH mode. I0520 05:31:49.933514 281472972518272 test_util.py:1494] Running testAddWithRefCountTwoAndNoDependence in GRAPH mode. INFO:tensorflow:time(__main__.FusedConv2DTest.testAddWithRefCountTwoAndNoDependence): 0.07s I0520 05:31:50.007266 281472972518272 test_util.py:2464] time(__main__.FusedConv2DTest.testAddWithRefCountTwoAndNoDependence): 0.07s INFO:tensorflow:Running testAddWithRefCountTwoAndNoDependence in EAGER mode. I0520 05:31:50.008720 281472972518272 test_util.py:1503] Running testAddWithRefCountTwoAndNoDependence in EAGER mode. INFO:tensorflow:time(__main__.FusedConv2DTest.testAddWithRefCountTwoAndNoDependence): 0.02s I0520 05:31:50.033498 281472972518272 test_util.py:2464] time(__main__.FusedConv2DTest.testAddWithRefCountTwoAndNoDependence): 0.02s [ OK ] FusedConv2DTest.testAddWithRefCountTwoAndNoDependence [ RUN ] FusedConv2DTest.testResizeAndPadLargeResize INFO:tensorflow:Running testResizeAndPadLargeResize in GRAPH mode. I0520 05:31:50.035042 281472972518272 test_util.py:1494] Running testResizeAndPadLargeResize in GRAPH mode. 2023-05-20 05:31:50.056072: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at conv_ops_fused_image_transform.cc:671 : INVALID_ARGUMENT: Encountered overflow when multiplying 5637144576 with 1879048192, result: -7854277750134145024 2023-05-20 05:31:50.056203: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: Encountered overflow when multiplying 5637144576 with 1879048192, result: -7854277750134145024 [[{{node FusedResizeAndPadConv2D}}]] ERROR:tensorflow:Graph execution error: Detected at node 'FusedResizeAndPadConv2D' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3504, in test.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.10/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.10/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.10/unittest/runner.py", line 184, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/case.py", line 650, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/case.py", line 591, in run self._callTestMethod(testMethod) File "/usr/lib/python3.10/unittest/case.py", line 549, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3449, in testResizeAndPadLargeResize gen_nn_ops.fused_resize_and_pad_conv2d( Node: 'FusedResizeAndPadConv2D' Encountered overflow when multiplying 5637144576 with 1879048192, result: -7854277750134145024 [[{{node FusedResizeAndPadConv2D}}]] Original stack trace for 'FusedResizeAndPadConv2D': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3504, in test.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.10/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.10/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.10/unittest/runner.py", line 184, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/case.py", line 650, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/case.py", line 591, in run self._callTestMethod(testMethod) File "/usr/lib/python3.10/unittest/case.py", line 549, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2335, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1498, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3449, in testResizeAndPadLargeResize gen_nn_ops.fused_resize_and_pad_conv2d( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_nn_ops.py", line 5132, in fused_resize_and_pad_conv2d _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3315, in _create_op_internal ret = Operation.from_node_def( E0520 05:31:50.706090 281472972518272 test_util.py:2067] Graph execution error: Detected at node 'FusedResizeAndPadConv2D' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3504, in test.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.10/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.10/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.10/unittest/runner.py", line 184, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/case.py", line 650, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/case.py", line 591, in run self._callTestMethod(testMethod) File "/usr/lib/python3.10/unittest/case.py", line 549, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3449, in testResizeAndPadLargeResize gen_nn_ops.fused_resize_and_pad_conv2d( Node: 'FusedResizeAndPadConv2D' Encountered overflow when multiplying 5637144576 with 1879048192, result: -7854277750134145024 [[{{node FusedResizeAndPadConv2D}}]] Original stack trace for 'FusedResizeAndPadConv2D': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3504, in test.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.10/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.10/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.10/unittest/runner.py", line 184, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/case.py", line 650, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/case.py", line 591, in run self._callTestMethod(testMethod) File "/usr/lib/python3.10/unittest/case.py", line 549, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2335, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1498, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3449, in testResizeAndPadLargeResize gen_nn_ops.fused_resize_and_pad_conv2d( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_nn_ops.py", line 5132, in fused_resize_and_pad_conv2d _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3315, in _create_op_internal ret = Operation.from_node_def( INFO:tensorflow:time(__main__.FusedConv2DTest.testResizeAndPadLargeResize): 0.67s I0520 05:31:50.708699 281472972518272 test_util.py:2464] time(__main__.FusedConv2DTest.testResizeAndPadLargeResize): 0.67s INFO:tensorflow:Running testResizeAndPadLargeResize in EAGER mode. I0520 05:31:50.709878 281472972518272 test_util.py:1503] Running testResizeAndPadLargeResize in EAGER mode. 2023-05-20 05:31:50.721615: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at conv_ops_fused_image_transform.cc:671 : INVALID_ARGUMENT: Encountered overflow when multiplying 5637144576 with 1879048192, result: -7854277750134145024 INFO:tensorflow:time(__main__.FusedConv2DTest.testResizeAndPadLargeResize): 0.01s I0520 05:31:50.722235 281472972518272 test_util.py:2464] time(__main__.FusedConv2DTest.testResizeAndPadLargeResize): 0.01s [ OK ] FusedConv2DTest.testResizeAndPadLargeResize [ RUN ] SeparableConv2DTest.testSeparableConv2DEqualInputOutputDepthNCHW INFO:tensorflow:time(__main__.SeparableConv2DTest.testSeparableConv2DEqualInputOutputDepthNCHW): 0.0s I0520 05:31:50.723869 281472972518272 test_util.py:2464] time(__main__.SeparableConv2DTest.testSeparableConv2DEqualInputOutputDepthNCHW): 0.0s [ OK ] SeparableConv2DTest.testSeparableConv2DEqualInputOutputDepthNCHW ====================================================================== FAIL: testConv2DGroupConvFwd (__main__.Conv2DTest) Conv2DTest.testConv2DGroupConvFwd ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2335, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1498, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 852, in testConv2DGroupConvFwd self._VerifyGroupConvFwd([10, 32, 32, 16], filter_dims, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 839, in _VerifyGroupConvFwd self.assertAllClose( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1539, in decorated return f(*args, **kwds) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3172, in assertAllClose self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3128, in _assertAllCloseRecursive self._assertArrayLikeAllClose( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3065, in _assertArrayLikeAllClose np.testing.assert_allclose( File "/usr/local/lib/python3.10/dist-packages/numpy/testing/_private/utils.py", line 1527, in assert_allclose assert_array_compare(compare, actual, desired, err_msg=str(err_msg), File "/usr/local/lib/python3.10/dist-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=1e-05, atol=1e-05 Mismatched value: a is different from b. not close where = (array([0, 0, 0, ..., 9, 9, 9]), array([0, 0, 0, ..., 7, 7, 7]), array([ 0, 0, 0, ..., 31, 31, 31]), array([ 0, 1, 2, ..., 29, 30, 31])) not close lhs = [1.0729360e+06 1.5484760e+06 1.6210520e+06 ... 3.9282451e+08 3.9286291e+08 2.2004294e+08] not close rhs = [7.2055200e+05 1.0859000e+06 1.1446520e+06 ... 5.3390678e+08 5.3395901e+08 3.4552045e+08] not close dif = [3.5238400e+05 4.6257600e+05 4.7640000e+05 ... 1.4108227e+08 1.4109610e+08 1.2547750e+08] not close tol = [ 7.2055297 10.859009 11.446529 ... 5339.068 5339.59 3455.2043 ] dtype = float32, shape = (10, 8, 32, 32) Mismatched elements: 81920 / 81920 (100%) Max absolute difference: 1.410961e+08 Max relative difference: 0.4890473 x: array([[[[1.072936e+06, 1.548476e+06, 1.621052e+06, ..., 3.580604e+06, 3.653180e+06, 2.265128e+06], [2.758524e+06, 3.885114e+06, 3.966330e+06, ..., 6.159162e+06,... y: array([[[[7.205520e+05, 1.085900e+06, 1.144652e+06, ..., 2.730956e+06, 2.789708e+06, 1.830760e+06], [2.017836e+06, 2.994042e+06, 3.075258e+06, ..., 5.268090e+06,... ---------------------------------------------------------------------- Ran 84 tests in 19.344s FAILED (failures=1, skipped=2) ================================================================================ ==================== Test output for //tensorflow/python/kernel_tests/nn_ops:conv_ops_test_cpu (shard 3 of 4): 2023-05-20 05:31:52.596665: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. Running tests under Python 3.10.11: /usr/local/bin/python3 [ RUN ] Conv2DTest.testConv2D1x1Padding INFO:tensorflow:Running testConv2D1x1Padding in GRAPH mode. I0520 05:31:56.224607 281473412068224 test_util.py:1494] Running testConv2D1x1Padding in GRAPH mode. WARNING:tensorflow:From /usr/lib/python3.10/contextlib.py:103: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `self.session()` or `self.cached_session()` instead. W0520 05:31:56.225193 281473412068224 deprecation.py:364] From /usr/lib/python3.10/contextlib.py:103: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `self.session()` or `self.cached_session()` instead. 2023-05-20 05:31:56.332210: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:375] MLIR V1 optimization pass is not enabled WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py:154: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. W0520 05:31:56.373904 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py:154: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. 2023-05-20 05:31:56.374639: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D1x1Padding): 0.34s I0520 05:31:56.568065 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2D1x1Padding): 0.34s INFO:tensorflow:Running testConv2D1x1Padding in EAGER mode. I0520 05:31:56.570251 281473412068224 test_util.py:1503] Running testConv2D1x1Padding in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D1x1Padding): 0.16s I0520 05:31:56.732445 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2D1x1Padding): 0.16s [ OK ] Conv2DTest.testConv2D1x1Padding [ RUN ] Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput INFO:tensorflow:Running testConv2D2x2Depth1Padding0x0BackpropInput in GRAPH mode. I0520 05:31:56.734396 281473412068224 test_util.py:1494] Running testConv2D2x2Depth1Padding0x0BackpropInput in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput): 0.17s I0520 05:31:56.900513 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput): 0.17s INFO:tensorflow:Running testConv2D2x2Depth1Padding0x0BackpropInput in EAGER mode. I0520 05:31:56.902343 281473412068224 test_util.py:1503] Running testConv2D2x2Depth1Padding0x0BackpropInput in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput): 0.08s I0520 05:31:56.984557 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput): 0.08s [ OK ] Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput [ RUN ] Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput INFO:tensorflow:Running testConv2D2x2Depth1Padding2x2BackpropInput in GRAPH mode. I0520 05:31:56.986329 281473412068224 test_util.py:1494] Running testConv2D2x2Depth1Padding2x2BackpropInput in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput): 0.06s I0520 05:31:57.047754 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput): 0.06s INFO:tensorflow:Running testConv2D2x2Depth1Padding2x2BackpropInput in EAGER mode. I0520 05:31:57.049240 281473412068224 test_util.py:1503] Running testConv2D2x2Depth1Padding2x2BackpropInput in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput): 0.03s I0520 05:31:57.083066 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput): 0.03s [ OK ] Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput [ RUN ] Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput INFO:tensorflow:Running testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput in GRAPH mode. I0520 05:31:57.084716 281473412068224 test_util.py:1494] Running testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput): 0.06s I0520 05:31:57.142635 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput): 0.06s INFO:tensorflow:Running testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput in EAGER mode. I0520 05:31:57.144090 281473412068224 test_util.py:1503] Running testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput): 0.03s I0520 05:31:57.177324 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput): 0.03s [ OK ] Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput [ RUN ] Conv2DTest.testConv2D2x2Depth1ValidBackpropInputDilation1x2 INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1ValidBackpropInputDilation1x2): 0.11s I0520 05:31:57.288108 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1ValidBackpropInputDilation1x2): 0.11s [ OK ] Conv2DTest.testConv2D2x2Depth1ValidBackpropInputDilation1x2 [ RUN ] Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2 INFO:tensorflow:Running testConv2D2x2Depth3ValidBackpropFilterStride1x2 in GRAPH mode. I0520 05:31:57.289992 281473412068224 test_util.py:1494] Running testConv2D2x2Depth3ValidBackpropFilterStride1x2 in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2): 0.06s I0520 05:31:57.348493 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2): 0.06s INFO:tensorflow:Running testConv2D2x2Depth3ValidBackpropFilterStride1x2 in EAGER mode. I0520 05:31:57.349993 281473412068224 test_util.py:1503] Running testConv2D2x2Depth3ValidBackpropFilterStride1x2 in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2): 0.03s I0520 05:31:57.378016 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2): 0.03s [ OK ] Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2 [ RUN ] Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2 INFO:tensorflow:Running testConv2D2x2Depth3ValidBackpropInputStride1x2 in GRAPH mode. I0520 05:31:57.379436 281473412068224 test_util.py:1494] Running testConv2D2x2Depth3ValidBackpropInputStride1x2 in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2): 0.03s I0520 05:31:57.408443 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2): 0.03s INFO:tensorflow:Running testConv2D2x2Depth3ValidBackpropInputStride1x2 in EAGER mode. I0520 05:31:57.409861 281473412068224 test_util.py:1503] Running testConv2D2x2Depth3ValidBackpropInputStride1x2 in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2): 0.02s I0520 05:31:57.427087 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2): 0.02s [ OK ] Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2 [ RUN ] Conv2DTest.testConv2D2x2FilterStride1x2 INFO:tensorflow:Running testConv2D2x2FilterStride1x2 in GRAPH mode. I0520 05:31:57.428438 281473412068224 test_util.py:1494] Running testConv2D2x2FilterStride1x2 in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2FilterStride1x2): 0.08s I0520 05:31:57.508238 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2FilterStride1x2): 0.08s INFO:tensorflow:Running testConv2D2x2FilterStride1x2 in EAGER mode. I0520 05:31:57.509916 281473412068224 test_util.py:1503] Running testConv2D2x2FilterStride1x2 in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2FilterStride1x2): 0.04s I0520 05:31:57.553165 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2FilterStride1x2): 0.04s [ OK ] Conv2DTest.testConv2D2x2FilterStride1x2 [ RUN ] Conv2DTest.testConv2DBackpropFilterWithEmptyInput INFO:tensorflow:Running testConv2DBackpropFilterWithEmptyInput in GRAPH mode. I0520 05:31:57.554697 281473412068224 test_util.py:1494] Running testConv2DBackpropFilterWithEmptyInput in GRAPH mode. 2023-05-20 05:31:57.599374: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2DBackpropFilter" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 1 } } } } inputs { dtype: DT_INT32 shape { dim { size: 4 } } value { dtype: DT_INT32 tensor_shape { dim { size: 4 } } tensor_content: "\002\000\000\000\002\000\000\000\001\000\000\000\001\000\000\000" } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 1 } dim { size: 2 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 1 } dim { size: 2 } dim { size: 1 } } } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: 2 } dim { size: 2 } dim { size: 1 } dim { size: 1 } } } 2023-05-20 05:31:57.648368: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2DBackpropFilter" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 1 } } } } inputs { dtype: DT_INT32 shape { dim { size: 4 } } value { dtype: DT_INT32 tensor_shape { dim { size: 4 } } tensor_content: "\002\000\000\000\002\000\000\000\001\000\000\000\001\000\000\000" } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 1 } dim { size: 2 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 1 } dim { size: 2 } dim { size: 1 } } } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: 2 } dim { size: 2 } dim { size: 1 } dim { size: 1 } } } INFO:tensorflow:time(__main__.Conv2DTest.testConv2DBackpropFilterWithEmptyInput): 0.11s I0520 05:31:57.668248 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2DBackpropFilterWithEmptyInput): 0.11s INFO:tensorflow:Running testConv2DBackpropFilterWithEmptyInput in EAGER mode. I0520 05:31:57.669799 281473412068224 test_util.py:1503] Running testConv2DBackpropFilterWithEmptyInput in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DBackpropFilterWithEmptyInput): 0.01s I0520 05:31:57.683857 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2DBackpropFilterWithEmptyInput): 0.01s [ OK ] Conv2DTest.testConv2DBackpropFilterWithEmptyInput [ RUN ] Conv2DTest.testConv2DEmptyBackpropFilter INFO:tensorflow:Running testConv2DEmptyBackpropFilter in GRAPH mode. I0520 05:31:57.685226 281473412068224 test_util.py:1494] Running testConv2DEmptyBackpropFilter in GRAPH mode. 2023-05-20 05:31:57.715508: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2DBackpropFilter" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 2 } dim { size: 3 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 2 } dim { size: 3 } dim { size: 1 } } tensor_content: "\000\000\200?\000\000\000@\000\000@@\000\000\200@\000\000\240@\000\000\300@" } } inputs { dtype: DT_INT32 shape { dim { size: 4 } } value { dtype: DT_INT32 tensor_shape { dim { size: 4 } } tensor_content: "\002\000\000\000\002\000\000\000\001\000\000\000\000\000\000\000" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1 } dim { size: 2 } dim { } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 1 } dim { size: 2 } dim { } } } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: 2 } dim { size: 2 } dim { size: 1 } dim { } } } 2023-05-20 05:31:57.747145: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2DBackpropFilter" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 2 } dim { size: 3 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 2 } dim { size: 3 } dim { size: 1 } } tensor_content: "\000\000\200?\000\000\000@\000\000@@\000\000\200@\000\000\240@\000\000\300@" } } inputs { dtype: DT_INT32 shape { dim { size: 4 } } value { dtype: DT_INT32 tensor_shape { dim { size: 4 } } tensor_content: "\002\000\000\000\002\000\000\000\001\000\000\000\000\000\000\000" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1 } dim { size: 2 } dim { } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 1 } dim { size: 2 } dim { } } } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: 2 } dim { size: 2 } dim { size: 1 } dim { } } } INFO:tensorflow:time(__main__.Conv2DTest.testConv2DEmptyBackpropFilter): 0.08s I0520 05:31:57.762805 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2DEmptyBackpropFilter): 0.08s INFO:tensorflow:Running testConv2DEmptyBackpropFilter in EAGER mode. I0520 05:31:57.764304 281473412068224 test_util.py:1503] Running testConv2DEmptyBackpropFilter in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DEmptyBackpropFilter): 0.01s I0520 05:31:57.778811 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2DEmptyBackpropFilter): 0.01s [ OK ] Conv2DTest.testConv2DEmptyBackpropFilter [ RUN ] Conv2DTest.testConv2DEmptyDilation INFO:tensorflow:Running testConv2DEmptyDilation in GRAPH mode. I0520 05:31:57.780216 281473412068224 test_util.py:1494] Running testConv2DEmptyDilation in GRAPH mode. 2023-05-20 05:31:57.831718: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2D" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 1 } dim { size: 3 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } tensor_content: "\000\000\200?\000\000\000@\000\000@@\000\000\200@\000\000\240@\000\000\300@\000\000\340@\000\000\000A\000\000\020A" } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { } dim { size: 1 } dim { size: 3 } dim { size: 3 } } } 2023-05-20 05:31:57.849242: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2D" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 2 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } tensor_content: "\000\000\200?\000\000\000@\000\000@@\000\000\200@\000\000\240@\000\000\300@\000\000\340@\000\000\000A\000\000\020A" } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } } 2023-05-20 05:31:57.849631: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2D" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 2 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } tensor_content: "\000\000\200?\000\000\000@\000\000@@\000\000\200@\000\000\240@\000\000\300@\000\000\340@\000\000\000A\000\000\020A" } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } } INFO:tensorflow:time(__main__.Conv2DTest.testConv2DEmptyDilation): 0.08s I0520 05:31:57.862060 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2DEmptyDilation): 0.08s INFO:tensorflow:Running testConv2DEmptyDilation in EAGER mode. I0520 05:31:57.863679 281473412068224 test_util.py:1503] Running testConv2DEmptyDilation in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DEmptyDilation): 0.03s I0520 05:31:57.895081 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2DEmptyDilation): 0.03s [ OK ] Conv2DTest.testConv2DEmptyDilation [ RUN ] Conv2DTest.testConv2DGroupConvFwd INFO:tensorflow:Running testConv2DGroupConvFwd in GRAPH mode. I0520 05:31:57.896745 281473412068224 test_util.py:1494] Running testConv2DGroupConvFwd in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DGroupConvFwd): 1.11s I0520 05:31:59.005520 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2DGroupConvFwd): 1.11s [ FAILED ] Conv2DTest.testConv2DGroupConvFwd [ RUN ] Conv2DTest.testConv2DKernelSizeMatchesInputSizeBackpropFilterDilation2x2 INFO:tensorflow:time(__main__.Conv2DTest.testConv2DKernelSizeMatchesInputSizeBackpropFilterDilation2x2): 0.09s I0520 05:31:59.104859 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2DKernelSizeMatchesInputSizeBackpropFilterDilation2x2): 0.09s [ OK ] Conv2DTest.testConv2DKernelSizeMatchesInputSizeBackpropFilterDilation2x2 [ RUN ] Conv2DTest.testConv2DKernelSmallerThanStrideSame INFO:tensorflow:Running testConv2DKernelSmallerThanStrideSame in GRAPH mode. I0520 05:31:59.107010 281473412068224 test_util.py:1494] Running testConv2DKernelSmallerThanStrideSame in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DKernelSmallerThanStrideSame): 0.24s I0520 05:31:59.342153 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2DKernelSmallerThanStrideSame): 0.24s INFO:tensorflow:Running testConv2DKernelSmallerThanStrideSame in EAGER mode. I0520 05:31:59.344737 281473412068224 test_util.py:1503] Running testConv2DKernelSmallerThanStrideSame in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DKernelSmallerThanStrideSame): 0.07s I0520 05:31:59.411139 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2DKernelSmallerThanStrideSame): 0.07s [ OK ] Conv2DTest.testConv2DKernelSmallerThanStrideSame [ RUN ] Conv2DTest.testConv2DOnlyTopRightPadding INFO:tensorflow:Running testConv2DOnlyTopRightPadding in GRAPH mode. I0520 05:31:59.412925 281473412068224 test_util.py:1494] Running testConv2DOnlyTopRightPadding in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DOnlyTopRightPadding): 0.21s I0520 05:31:59.626335 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2DOnlyTopRightPadding): 0.21s INFO:tensorflow:Running testConv2DOnlyTopRightPadding in EAGER mode. I0520 05:31:59.628784 281473412068224 test_util.py:1503] Running testConv2DOnlyTopRightPadding in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DOnlyTopRightPadding): 0.09s I0520 05:31:59.720679 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConv2DOnlyTopRightPadding): 0.09s [ OK ] Conv2DTest.testConv2DOnlyTopRightPadding [ RUN ] Conv2DTest.testConvolutionClass2DExpandedBatch INFO:tensorflow:Running testConvolutionClass2DExpandedBatch in GRAPH mode. I0520 05:31:59.723122 281473412068224 test_util.py:1494] Running testConvolutionClass2DExpandedBatch in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConvolutionClass2DExpandedBatch): 0.07s I0520 05:31:59.797333 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConvolutionClass2DExpandedBatch): 0.07s INFO:tensorflow:Running testConvolutionClass2DExpandedBatch in EAGER mode. I0520 05:31:59.799139 281473412068224 test_util.py:1503] Running testConvolutionClass2DExpandedBatch in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConvolutionClass2DExpandedBatch): 0.04s I0520 05:31:59.838943 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testConvolutionClass2DExpandedBatch): 0.04s [ OK ] Conv2DTest.testConvolutionClass2DExpandedBatch [ RUN ] Conv2DTest.testFilterGradient1x1PaddingStrideOne WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py:1953: compute_gradient (from tensorflow.python.ops.gradient_checker) is deprecated and will be removed in a future version. Instructions for updating: Use tf.test.compute_gradient in 2.0, which has better support for functions. Note that the two versions have different usage, so code change is needed. W0520 05:31:59.848102 281473412068224 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py:1953: compute_gradient (from tensorflow.python.ops.gradient_checker) is deprecated and will be removed in a future version. Instructions for updating: Use tf.test.compute_gradient in 2.0, which has better support for functions. Note that the two versions have different usage, so code change is needed. INFO:tensorflow:time(__main__.Conv2DTest.testFilterGradient1x1PaddingStrideOne): 0.63s I0520 05:32:00.469917 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testFilterGradient1x1PaddingStrideOne): 0.63s [ OK ] Conv2DTest.testFilterGradient1x1PaddingStrideOne [ RUN ] Conv2DTest.testFilterGradientGroupConv INFO:tensorflow:time(__main__.Conv2DTest.testFilterGradientGroupConv): 0.0s I0520 05:32:00.472221 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testFilterGradientGroupConv): 0.0s [ SKIPPED ] Conv2DTest.testFilterGradientGroupConv [ RUN ] Conv2DTest.testFilterGradientSamePaddingStrideThree INFO:tensorflow:time(__main__.Conv2DTest.testFilterGradientSamePaddingStrideThree): 1.35s I0520 05:32:01.824566 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testFilterGradientSamePaddingStrideThree): 1.35s [ OK ] Conv2DTest.testFilterGradientSamePaddingStrideThree [ RUN ] Conv2DTest.testFilterGradientValidPaddingStrideTwo INFO:tensorflow:time(__main__.Conv2DTest.testFilterGradientValidPaddingStrideTwo): 1.26s I0520 05:32:03.085258 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testFilterGradientValidPaddingStrideTwo): 1.26s [ OK ] Conv2DTest.testFilterGradientValidPaddingStrideTwo [ RUN ] Conv2DTest.testInceptionBackFilter_11 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:03.087224 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], [1, 1], 'SAME') I0520 05:32:03.088460 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_11): 0.07s I0520 05:32:03.158569 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_11): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:03.160079 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], [1, 1], 'SAME') I0520 05:32:03.160387 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_11): 0.03s I0520 05:32:03.187804 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_11): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_11 [ RUN ] Conv2DTest.testInceptionBackFilter_15 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:03.189194 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], [2, 2], 'VALID') I0520 05:32:03.190506 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], [2, 2], 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_15): 0.06s I0520 05:32:03.249533 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_15): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:03.251002 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], [2, 2], 'VALID') I0520 05:32:03.251319 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], [2, 2], 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_15): 0.02s I0520 05:32:03.275267 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_15): 0.02s [ OK ] Conv2DTest.testInceptionBackFilter_15 [ RUN ] Conv2DTest.testInceptionBackFilter_19 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:03.276654 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], [1, 1], 'SAME') I0520 05:32:03.278018 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_19): 0.07s I0520 05:32:03.343215 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_19): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:03.344765 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], [1, 1], 'SAME') I0520 05:32:03.345091 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_19): 0.03s I0520 05:32:03.370038 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_19): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_19 [ RUN ] Conv2DTest.testInceptionBackFilter_22 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:03.371455 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], [1, 1], 'SAME') I0520 05:32:03.372803 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_22): 0.08s I0520 05:32:03.447203 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_22): 0.08s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:03.448979 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], [1, 1], 'SAME') I0520 05:32:03.449300 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_22): 0.04s I0520 05:32:03.491410 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_22): 0.04s [ OK ] Conv2DTest.testInceptionBackFilter_22 [ RUN ] Conv2DTest.testInceptionBackFilter_26 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:03.492943 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], [1, 1], 'SAME') I0520 05:32:03.494363 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_26): 0.1s I0520 05:32:03.592041 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_26): 0.1s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:03.593821 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], [1, 1], 'SAME') I0520 05:32:03.594145 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_26): 0.04s I0520 05:32:03.632054 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_26): 0.04s [ OK ] Conv2DTest.testInceptionBackFilter_26 [ RUN ] Conv2DTest.testInceptionBackFilter_3 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:03.633611 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], [1, 1], 'SAME') I0520 05:32:03.635134 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_3): 0.08s I0520 05:32:03.713509 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_3): 0.08s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:03.715294 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], [1, 1], 'SAME') I0520 05:32:03.715609 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_3): 0.03s I0520 05:32:03.743533 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_3): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_3 [ RUN ] Conv2DTest.testInceptionBackFilter_33 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:03.745148 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], [1, 1], 'SAME') I0520 05:32:03.746652 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_33): 0.07s I0520 05:32:03.816815 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_33): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:03.818670 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], [1, 1], 'SAME') I0520 05:32:03.819017 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_33): 0.03s I0520 05:32:03.852724 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_33): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_33 [ RUN ] Conv2DTest.testInceptionBackFilter_37 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:03.854412 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], [1, 1], 'SAME') I0520 05:32:03.856041 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_37): 0.07s I0520 05:32:03.928219 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_37): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:03.930055 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], [1, 1], 'SAME') I0520 05:32:03.930416 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_37): 0.03s I0520 05:32:03.961279 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_37): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_37 [ RUN ] Conv2DTest.testInceptionBackFilter_40 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:03.962929 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], [1, 1], 'SAME') I0520 05:32:03.964480 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_40): 0.1s I0520 05:32:04.062203 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_40): 0.1s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:04.064072 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], [1, 1], 'SAME') I0520 05:32:04.064419 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_40): 0.04s I0520 05:32:04.099539 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_40): 0.04s [ OK ] Conv2DTest.testInceptionBackFilter_40 [ RUN ] Conv2DTest.testInceptionBackFilter_44 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:04.101169 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], [1, 1], 'SAME') I0520 05:32:04.102751 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_44): 0.11s I0520 05:32:04.213054 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_44): 0.11s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:04.214887 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], [1, 1], 'SAME') I0520 05:32:04.215238 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_44): 0.03s I0520 05:32:04.248100 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_44): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_44 [ RUN ] Conv2DTest.testInceptionBackFilter_48 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:04.249656 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], [1, 1], 'SAME') I0520 05:32:04.251074 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_48): 0.12s I0520 05:32:04.372179 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_48): 0.12s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:04.374051 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], [1, 1], 'SAME') I0520 05:32:04.374372 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_48): 0.05s I0520 05:32:04.428032 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_48): 0.05s [ OK ] Conv2DTest.testInceptionBackFilter_48 [ RUN ] Conv2DTest.testInceptionBackFilter_51 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:04.429643 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], [1, 1], 'SAME') I0520 05:32:04.431201 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_51): 0.1s I0520 05:32:04.525632 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_51): 0.1s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:04.527292 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], [1, 1], 'SAME') I0520 05:32:04.527656 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_51): 0.03s I0520 05:32:04.557780 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_51): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_51 [ RUN ] Conv2DTest.testInceptionBackFilter_6 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:04.559278 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], [1, 1], 'SAME') I0520 05:32:04.560737 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_6): 0.08s I0520 05:32:04.637965 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_6): 0.08s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:04.639789 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], [1, 1], 'SAME') I0520 05:32:04.640116 281473412068224 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_6): 0.03s I0520 05:32:04.669621 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_6): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_6 [ RUN ] Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:04.671246 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Skipping InceptionBackFilter ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], [1, 1], 'SAME') I0520 05:32:04.672842 281473412068224 conv_ops_test.py:3254] Skipping InceptionBackFilter ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused): 0.0s I0520 05:32:04.673261 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused): 0.0s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:04.674119 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Skipping InceptionBackFilter ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], [1, 1], 'SAME') I0520 05:32:04.674466 281473412068224 conv_ops_test.py:3254] Skipping InceptionBackFilter ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused): 0.0s I0520 05:32:04.674866 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused): 0.0s [ OK ] Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused [ RUN ] Conv2DTest.testInceptionBackInput_11 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:04.675764 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], 1, 'SAME') I0520 05:32:04.676972 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_11): 0.06s I0520 05:32:04.738772 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_11): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:04.740525 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], 1, 'SAME') I0520 05:32:04.740856 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_11): 0.07s I0520 05:32:04.812236 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_11): 0.07s [ OK ] Conv2DTest.testInceptionBackInput_11 [ RUN ] Conv2DTest.testInceptionBackInput_15 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:04.813472 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], 2, 'VALID') I0520 05:32:04.814502 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], 2, 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_15): 0.07s I0520 05:32:04.882121 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_15): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:04.883541 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], 2, 'VALID') I0520 05:32:04.883760 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], 2, 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_15): 0.02s I0520 05:32:04.903083 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_15): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_15 [ RUN ] Conv2DTest.testInceptionBackInput_19 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:04.904548 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], 1, 'SAME') I0520 05:32:04.905951 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_19): 0.07s I0520 05:32:04.974014 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_19): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:04.975890 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], 1, 'SAME') I0520 05:32:04.976265 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_19): 0.02s I0520 05:32:04.997952 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_19): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_19 [ RUN ] Conv2DTest.testInceptionBackInput_22 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:04.999490 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], 1, 'SAME') I0520 05:32:05.000953 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_22): 0.07s I0520 05:32:05.067121 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_22): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:05.068902 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], 1, 'SAME') I0520 05:32:05.069245 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_22): 0.02s I0520 05:32:05.091145 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_22): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_22 [ RUN ] Conv2DTest.testInceptionBackInput_26 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:05.092669 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], 1, 'SAME') I0520 05:32:05.094104 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_26): 0.07s I0520 05:32:05.158486 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_26): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:05.160335 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], 1, 'SAME') I0520 05:32:05.160695 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_26): 0.03s I0520 05:32:05.185606 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_26): 0.03s [ OK ] Conv2DTest.testInceptionBackInput_26 [ RUN ] Conv2DTest.testInceptionBackInput_3 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:05.187249 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], 1, 'SAME') I0520 05:32:05.188823 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_3): 0.06s I0520 05:32:05.250117 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_3): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:05.251824 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], 1, 'SAME') I0520 05:32:05.252106 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_3): 0.03s I0520 05:32:05.277307 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_3): 0.03s [ OK ] Conv2DTest.testInceptionBackInput_3 [ RUN ] Conv2DTest.testInceptionBackInput_33 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:05.279008 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], 1, 'SAME') I0520 05:32:05.280676 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_33): 0.3s I0520 05:32:05.578146 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_33): 0.3s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:05.579679 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], 1, 'SAME') I0520 05:32:05.579910 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_33): 0.09s I0520 05:32:05.668769 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_33): 0.09s [ OK ] Conv2DTest.testInceptionBackInput_33 [ RUN ] Conv2DTest.testInceptionBackInput_37 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:05.670331 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], 1, 'SAME') I0520 05:32:05.671861 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_37): 0.14s I0520 05:32:05.810791 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_37): 0.14s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:05.812632 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], 1, 'SAME') I0520 05:32:05.812984 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_37): 0.02s I0520 05:32:05.835251 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_37): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_37 [ RUN ] Conv2DTest.testInceptionBackInput_40 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:05.837026 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], 1, 'SAME') I0520 05:32:05.838629 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_40): 0.08s I0520 05:32:05.917495 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_40): 0.08s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:05.919502 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], 1, 'SAME') I0520 05:32:05.919892 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_40): 0.03s I0520 05:32:05.947055 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_40): 0.03s [ OK ] Conv2DTest.testInceptionBackInput_40 [ RUN ] Conv2DTest.testInceptionBackInput_44 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:05.948856 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], 1, 'SAME') I0520 05:32:05.950631 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_44): 0.16s I0520 05:32:06.106163 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_44): 0.16s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:06.108227 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], 1, 'SAME') I0520 05:32:06.108615 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_44): 0.04s I0520 05:32:06.149863 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_44): 0.04s [ OK ] Conv2DTest.testInceptionBackInput_44 [ RUN ] Conv2DTest.testInceptionBackInput_48 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:06.151705 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], 1, 'SAME') I0520 05:32:06.153568 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_48): 0.08s I0520 05:32:06.236310 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_48): 0.08s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:06.238237 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], 1, 'SAME') I0520 05:32:06.238606 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_48): 0.04s I0520 05:32:06.275276 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_48): 0.04s [ OK ] Conv2DTest.testInceptionBackInput_48 [ RUN ] Conv2DTest.testInceptionBackInput_51 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:06.276958 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], 1, 'SAME') I0520 05:32:06.278674 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_51): 0.07s I0520 05:32:06.349899 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_51): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:06.351687 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], 1, 'SAME') I0520 05:32:06.352020 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_51): 0.02s I0520 05:32:06.373257 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_51): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_51 [ RUN ] Conv2DTest.testInceptionBackInput_6 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:06.374714 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], 1, 'SAME') I0520 05:32:06.376185 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_6): 0.06s I0520 05:32:06.436414 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_6): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:06.438103 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], 1, 'SAME') I0520 05:32:06.438389 281473412068224 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_6): 0.02s I0520 05:32:06.458394 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_6): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_6 [ RUN ] Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:06.459791 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Skipping InceptionBackInput ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], 1, 'SAME') I0520 05:32:06.461281 281473412068224 conv_ops_test.py:3238] Skipping InceptionBackInput ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused): 0.0s I0520 05:32:06.461638 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused): 0.0s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:06.462396 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Skipping InceptionBackInput ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], 1, 'SAME') I0520 05:32:06.462697 281473412068224 conv_ops_test.py:3238] Skipping InceptionBackInput ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused): 0.0s I0520 05:32:06.463018 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused): 0.0s [ OK ] Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_11 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:06.463778 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') I0520 05:32:06.464710 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_11): 0.18s I0520 05:32:06.645626 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_11): 0.18s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:06.647311 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') I0520 05:32:06.647610 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_11): 0.04s I0520 05:32:06.685383 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_11): 0.04s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_11 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_15 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:06.687048 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_15): 0.0s I0520 05:32:06.688557 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_15): 0.0s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:06.689354 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_15): 0.0s I0520 05:32:06.689668 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_15): 0.0s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_15 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_19 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:06.690404 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') I0520 05:32:06.691375 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_19): 0.14s I0520 05:32:06.830879 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_19): 0.14s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:06.832602 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') I0520 05:32:06.832921 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_19): 0.04s I0520 05:32:06.876785 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_19): 0.04s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_19 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_22 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:06.878420 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') I0520 05:32:06.879869 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_22): 0.11s I0520 05:32:06.989601 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_22): 0.11s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:06.991397 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') I0520 05:32:06.991736 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_22): 0.08s I0520 05:32:07.074129 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_22): 0.08s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_22 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_26 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:07.075665 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') I0520 05:32:07.077081 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_26): 0.14s I0520 05:32:07.211437 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_26): 0.14s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:07.213038 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') I0520 05:32:07.213325 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_26): 0.04s I0520 05:32:07.253543 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_26): 0.04s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_26 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_3 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:07.255348 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') I0520 05:32:07.256829 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_3): 0.12s I0520 05:32:07.372634 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_3): 0.12s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:07.374476 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') I0520 05:32:07.374796 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_3): 0.04s I0520 05:32:07.414980 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_3): 0.04s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_3 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_33 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:07.416661 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') I0520 05:32:07.418106 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_33): 0.1s I0520 05:32:07.516794 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_33): 0.1s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:07.518597 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') I0520 05:32:07.518896 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_33): 0.05s I0520 05:32:07.564670 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_33): 0.05s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_33 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_37 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:07.566374 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') I0520 05:32:07.567779 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_37): 0.15s I0520 05:32:07.712116 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_37): 0.15s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:07.713898 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') I0520 05:32:07.714208 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_37): 0.05s I0520 05:32:07.767153 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_37): 0.05s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_37 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_40 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:07.768809 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') I0520 05:32:07.770185 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_40): 0.13s I0520 05:32:07.894621 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_40): 0.13s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:07.896414 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') I0520 05:32:07.896709 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_40): 0.04s I0520 05:32:07.940591 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_40): 0.04s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_40 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_44 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:07.942185 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') I0520 05:32:07.943547 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_44): 0.25s I0520 05:32:08.196888 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_44): 0.25s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:08.198601 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') I0520 05:32:08.198895 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_44): 0.06s I0520 05:32:08.256297 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_44): 0.06s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_44 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_48 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:08.258065 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') I0520 05:32:08.259576 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_48): 0.13s I0520 05:32:08.388611 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_48): 0.13s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:08.390649 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') I0520 05:32:08.390954 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_48): 0.06s I0520 05:32:08.451107 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_48): 0.06s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_48 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_51 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:08.452853 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') I0520 05:32:08.454275 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_51): 0.12s I0520 05:32:08.576503 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_51): 0.12s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:08.578665 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') I0520 05:32:08.579018 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_51): 0.06s I0520 05:32:08.637394 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_51): 0.06s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_51 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_6 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:08.638950 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') I0520 05:32:08.640145 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_6): 0.14s I0520 05:32:08.778442 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_6): 0.14s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:08.780321 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') I0520 05:32:08.780634 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_6): 0.05s I0520 05:32:08.829792 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_6): 0.05s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_6 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:08.831388 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') I0520 05:32:08.832769 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused): 1.79s I0520 05:32:10.617622 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused): 1.79s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:10.619544 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') I0520 05:32:10.619809 281473412068224 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused): 1.91s I0520 05:32:12.525357 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused): 1.91s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused [ RUN ] Conv2DTest.testInceptionFwd_11 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:12.527342 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') I0520 05:32:12.528720 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_11): 0.04s I0520 05:32:12.563184 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_11): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:12.564576 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') I0520 05:32:12.564806 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_11): 0.03s I0520 05:32:12.592102 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_11): 0.03s [ OK ] Conv2DTest.testInceptionFwd_11 [ RUN ] Conv2DTest.testInceptionFwd_15 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:12.593313 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [3, 3, 12, 32], 2, 'VALID') I0520 05:32:12.594388 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [3, 3, 12, 32], 2, 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_15): 0.05s I0520 05:32:12.639897 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_15): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:12.641244 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [3, 3, 12, 32], 2, 'VALID') I0520 05:32:12.641482 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [3, 3, 12, 32], 2, 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_15): 0.02s I0520 05:32:12.657985 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_15): 0.02s [ OK ] Conv2DTest.testInceptionFwd_15 [ RUN ] Conv2DTest.testInceptionFwd_19 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:12.659233 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') I0520 05:32:12.660317 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_19): 0.04s I0520 05:32:12.702400 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_19): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:12.703839 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') I0520 05:32:12.704123 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_19): 0.02s I0520 05:32:12.723312 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_19): 0.02s [ OK ] Conv2DTest.testInceptionFwd_19 [ RUN ] Conv2DTest.testInceptionFwd_22 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:12.724462 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') I0520 05:32:12.725399 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_22): 0.04s I0520 05:32:12.760280 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_22): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:12.761559 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') I0520 05:32:12.761752 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_22): 0.02s I0520 05:32:12.779819 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_22): 0.02s [ OK ] Conv2DTest.testInceptionFwd_22 [ RUN ] Conv2DTest.testInceptionFwd_26 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:12.780885 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') I0520 05:32:12.781820 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_26): 0.05s I0520 05:32:12.828846 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_26): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:12.830154 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') I0520 05:32:12.830353 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_26): 0.02s I0520 05:32:12.850426 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_26): 0.02s [ OK ] Conv2DTest.testInceptionFwd_26 [ RUN ] Conv2DTest.testInceptionFwd_3 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:12.851635 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') I0520 05:32:12.852754 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_3): 0.06s I0520 05:32:12.910343 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_3): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:12.911613 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') I0520 05:32:12.911800 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_3): 0.02s I0520 05:32:12.928282 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_3): 0.02s [ OK ] Conv2DTest.testInceptionFwd_3 [ RUN ] Conv2DTest.testInceptionFwd_33 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:12.929327 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') I0520 05:32:12.930237 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_33): 0.06s I0520 05:32:12.988010 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_33): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:12.989301 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') I0520 05:32:12.989480 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_33): 0.02s I0520 05:32:13.006524 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_33): 0.02s [ OK ] Conv2DTest.testInceptionFwd_33 [ RUN ] Conv2DTest.testInceptionFwd_37 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:13.007725 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') I0520 05:32:13.008857 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_37): 0.04s I0520 05:32:13.042815 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_37): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:13.044248 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') I0520 05:32:13.044498 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_37): 0.02s I0520 05:32:13.059923 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_37): 0.02s [ OK ] Conv2DTest.testInceptionFwd_37 [ RUN ] Conv2DTest.testInceptionFwd_40 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:13.061191 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') I0520 05:32:13.062336 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_40): 0.05s I0520 05:32:13.107579 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_40): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:13.108930 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') I0520 05:32:13.109163 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_40): 0.02s I0520 05:32:13.128602 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_40): 0.02s [ OK ] Conv2DTest.testInceptionFwd_40 [ RUN ] Conv2DTest.testInceptionFwd_44 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:13.129858 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') I0520 05:32:13.131025 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_44): 0.05s I0520 05:32:13.181567 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_44): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:13.182941 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') I0520 05:32:13.183182 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_44): 0.02s I0520 05:32:13.198610 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_44): 0.02s [ OK ] Conv2DTest.testInceptionFwd_44 [ RUN ] Conv2DTest.testInceptionFwd_48 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:13.199872 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') I0520 05:32:13.201045 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_48): 0.04s I0520 05:32:13.237126 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_48): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:13.238469 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') I0520 05:32:13.238716 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_48): 0.01s I0520 05:32:13.253267 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_48): 0.01s [ OK ] Conv2DTest.testInceptionFwd_48 [ RUN ] Conv2DTest.testInceptionFwd_51 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:13.254534 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') I0520 05:32:13.255682 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_51): 0.05s I0520 05:32:13.300871 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_51): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:13.302336 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') I0520 05:32:13.302646 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_51): 0.02s I0520 05:32:13.320275 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_51): 0.02s [ OK ] Conv2DTest.testInceptionFwd_51 [ RUN ] Conv2DTest.testInceptionFwd_6 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:13.321548 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') I0520 05:32:13.322633 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_6): 0.04s I0520 05:32:13.360110 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_6): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:13.361459 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') I0520 05:32:13.361712 281473412068224 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_6): 0.01s I0520 05:32:13.375806 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_6): 0.01s [ OK ] Conv2DTest.testInceptionFwd_6 [ RUN ] Conv2DTest.testInceptionFwd_No_Winograd_Nonfused INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:13.377145 281473412068224 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Skipping InceptionFwd ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') I0520 05:32:13.378480 281473412068224 conv_ops_test.py:3205] Skipping InceptionFwd ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_No_Winograd_Nonfused): 0.0s I0520 05:32:13.378837 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_No_Winograd_Nonfused): 0.0s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:13.379569 281473412068224 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Skipping InceptionFwd ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') I0520 05:32:13.379831 281473412068224 conv_ops_test.py:3205] Skipping InceptionFwd ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_No_Winograd_Nonfused): 0.0s I0520 05:32:13.380121 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_No_Winograd_Nonfused): 0.0s [ OK ] Conv2DTest.testInceptionFwd_No_Winograd_Nonfused [ RUN ] Conv2DTest.testInputGradient1x1PaddingStrideTwo INFO:tensorflow:time(__main__.Conv2DTest.testInputGradient1x1PaddingStrideTwo): 0.37s I0520 05:32:13.753826 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInputGradient1x1PaddingStrideTwo): 0.37s [ OK ] Conv2DTest.testInputGradient1x1PaddingStrideTwo [ RUN ] Conv2DTest.testInputGradientKernelSizeMatchesInputSize INFO:tensorflow:time(__main__.Conv2DTest.testInputGradientKernelSizeMatchesInputSize): 0.36s I0520 05:32:14.117010 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInputGradientKernelSizeMatchesInputSize): 0.36s [ OK ] Conv2DTest.testInputGradientKernelSizeMatchesInputSize [ RUN ] Conv2DTest.testInputGradientValidPaddingStrideOne INFO:tensorflow:time(__main__.Conv2DTest.testInputGradientValidPaddingStrideOne): 0.71s I0520 05:32:14.833654 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testInputGradientValidPaddingStrideOne): 0.71s [ OK ] Conv2DTest.testInputGradientValidPaddingStrideOne [ RUN ] Conv2DTest.testShapeFunctionEdgeCases INFO:tensorflow:time(__main__.Conv2DTest.testShapeFunctionEdgeCases): 0.02s I0520 05:32:14.859042 281473412068224 test_util.py:2464] time(__main__.Conv2DTest.testShapeFunctionEdgeCases): 0.02s [ OK ] Conv2DTest.testShapeFunctionEdgeCases [ RUN ] DeepConv2DTest.test_session [ SKIPPED ] DeepConv2DTest.test_session [ RUN ] FusedConv2DTest.testAddWithRefCountTwoAndNoDependence INFO:tensorflow:Running testAddWithRefCountTwoAndNoDependence in GRAPH mode. I0520 05:32:14.860338 281473412068224 test_util.py:1494] Running testAddWithRefCountTwoAndNoDependence in GRAPH mode. INFO:tensorflow:time(__main__.FusedConv2DTest.testAddWithRefCountTwoAndNoDependence): 0.05s I0520 05:32:14.911934 281473412068224 test_util.py:2464] time(__main__.FusedConv2DTest.testAddWithRefCountTwoAndNoDependence): 0.05s INFO:tensorflow:Running testAddWithRefCountTwoAndNoDependence in EAGER mode. I0520 05:32:14.913070 281473412068224 test_util.py:1503] Running testAddWithRefCountTwoAndNoDependence in EAGER mode. INFO:tensorflow:time(__main__.FusedConv2DTest.testAddWithRefCountTwoAndNoDependence): 0.03s I0520 05:32:14.939983 281473412068224 test_util.py:2464] time(__main__.FusedConv2DTest.testAddWithRefCountTwoAndNoDependence): 0.03s [ OK ] FusedConv2DTest.testAddWithRefCountTwoAndNoDependence [ RUN ] FusedConv2DTest.testResizeAndPadLargeResize INFO:tensorflow:Running testResizeAndPadLargeResize in GRAPH mode. I0520 05:32:14.941416 281473412068224 test_util.py:1494] Running testResizeAndPadLargeResize in GRAPH mode. 2023-05-20 05:32:14.962421: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at conv_ops_fused_image_transform.cc:671 : INVALID_ARGUMENT: Encountered overflow when multiplying 5637144576 with 1879048192, result: -7854277750134145024 2023-05-20 05:32:14.962480: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: Encountered overflow when multiplying 5637144576 with 1879048192, result: -7854277750134145024 [[{{node FusedResizeAndPadConv2D}}]] ERROR:tensorflow:Graph execution error: Detected at node 'FusedResizeAndPadConv2D' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3504, in test.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.10/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.10/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.10/unittest/runner.py", line 184, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/case.py", line 650, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/case.py", line 591, in run self._callTestMethod(testMethod) File "/usr/lib/python3.10/unittest/case.py", line 549, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3449, in testResizeAndPadLargeResize gen_nn_ops.fused_resize_and_pad_conv2d( Node: 'FusedResizeAndPadConv2D' Encountered overflow when multiplying 5637144576 with 1879048192, result: -7854277750134145024 [[{{node FusedResizeAndPadConv2D}}]] Original stack trace for 'FusedResizeAndPadConv2D': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3504, in test.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.10/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.10/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.10/unittest/runner.py", line 184, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/case.py", line 650, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/case.py", line 591, in run self._callTestMethod(testMethod) File "/usr/lib/python3.10/unittest/case.py", line 549, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2335, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1498, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3449, in testResizeAndPadLargeResize gen_nn_ops.fused_resize_and_pad_conv2d( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_nn_ops.py", line 5132, in fused_resize_and_pad_conv2d _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3315, in _create_op_internal ret = Operation.from_node_def( E0520 05:32:16.036830 281473412068224 test_util.py:2067] Graph execution error: Detected at node 'FusedResizeAndPadConv2D' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3504, in test.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.10/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.10/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.10/unittest/runner.py", line 184, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/case.py", line 650, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/case.py", line 591, in run self._callTestMethod(testMethod) File "/usr/lib/python3.10/unittest/case.py", line 549, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3449, in testResizeAndPadLargeResize gen_nn_ops.fused_resize_and_pad_conv2d( Node: 'FusedResizeAndPadConv2D' Encountered overflow when multiplying 5637144576 with 1879048192, result: -7854277750134145024 [[{{node FusedResizeAndPadConv2D}}]] Original stack trace for 'FusedResizeAndPadConv2D': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3504, in test.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.10/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.10/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.10/unittest/runner.py", line 184, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/case.py", line 650, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/case.py", line 591, in run self._callTestMethod(testMethod) File "/usr/lib/python3.10/unittest/case.py", line 549, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2335, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1498, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3449, in testResizeAndPadLargeResize gen_nn_ops.fused_resize_and_pad_conv2d( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_nn_ops.py", line 5132, in fused_resize_and_pad_conv2d _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3315, in _create_op_internal ret = Operation.from_node_def( INFO:tensorflow:time(__main__.FusedConv2DTest.testResizeAndPadLargeResize): 1.1s I0520 05:32:16.039096 281473412068224 test_util.py:2464] time(__main__.FusedConv2DTest.testResizeAndPadLargeResize): 1.1s INFO:tensorflow:Running testResizeAndPadLargeResize in EAGER mode. I0520 05:32:16.040276 281473412068224 test_util.py:1503] Running testResizeAndPadLargeResize in EAGER mode. 2023-05-20 05:32:16.050873: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at conv_ops_fused_image_transform.cc:671 : INVALID_ARGUMENT: Encountered overflow when multiplying 5637144576 with 1879048192, result: -7854277750134145024 INFO:tensorflow:time(__main__.FusedConv2DTest.testResizeAndPadLargeResize): 0.01s I0520 05:32:16.051467 281473412068224 test_util.py:2464] time(__main__.FusedConv2DTest.testResizeAndPadLargeResize): 0.01s [ OK ] FusedConv2DTest.testResizeAndPadLargeResize [ RUN ] SeparableConv2DTest.testSeparableConv2DEqualInputOutputDepthNCHW INFO:tensorflow:time(__main__.SeparableConv2DTest.testSeparableConv2DEqualInputOutputDepthNCHW): 0.0s I0520 05:32:16.052929 281473412068224 test_util.py:2464] time(__main__.SeparableConv2DTest.testSeparableConv2DEqualInputOutputDepthNCHW): 0.0s [ OK ] SeparableConv2DTest.testSeparableConv2DEqualInputOutputDepthNCHW ====================================================================== FAIL: testConv2DGroupConvFwd (__main__.Conv2DTest) Conv2DTest.testConv2DGroupConvFwd ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2335, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1498, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 852, in testConv2DGroupConvFwd self._VerifyGroupConvFwd([10, 32, 32, 16], filter_dims, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 839, in _VerifyGroupConvFwd self.assertAllClose( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1539, in decorated return f(*args, **kwds) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3172, in assertAllClose self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3128, in _assertAllCloseRecursive self._assertArrayLikeAllClose( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3065, in _assertArrayLikeAllClose np.testing.assert_allclose( File "/usr/local/lib/python3.10/dist-packages/numpy/testing/_private/utils.py", line 1527, in assert_allclose assert_array_compare(compare, actual, desired, err_msg=str(err_msg), File "/usr/local/lib/python3.10/dist-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=1e-05, atol=1e-05 Mismatched value: a is different from b. not close where = (array([0, 0, 0, ..., 9, 9, 9]), array([0, 0, 0, ..., 7, 7, 7]), array([ 0, 0, 0, ..., 31, 31, 31]), array([ 0, 1, 2, ..., 29, 30, 31])) not close lhs = [1.0729360e+06 1.5484760e+06 1.6210520e+06 ... 3.9282451e+08 3.9286291e+08 2.2004294e+08] not close rhs = [7.2055200e+05 1.0859000e+06 1.1446520e+06 ... 5.3390678e+08 5.3395901e+08 3.4552045e+08] not close dif = [3.5238400e+05 4.6257600e+05 4.7640000e+05 ... 1.4108227e+08 1.4109610e+08 1.2547750e+08] not close tol = [ 7.2055297 10.859009 11.446529 ... 5339.068 5339.59 3455.2043 ] dtype = float32, shape = (10, 8, 32, 32) Mismatched elements: 81920 / 81920 (100%) Max absolute difference: 1.410961e+08 Max relative difference: 0.4890473 x: array([[[[1.072936e+06, 1.548476e+06, 1.621052e+06, ..., 3.580604e+06, 3.653180e+06, 2.265128e+06], [2.758524e+06, 3.885114e+06, 3.966330e+06, ..., 6.159162e+06,... y: array([[[[7.205520e+05, 1.085900e+06, 1.144652e+06, ..., 2.730956e+06, 2.789708e+06, 1.830760e+06], [2.017836e+06, 2.994042e+06, 3.075258e+06, ..., 5.268090e+06,... ---------------------------------------------------------------------- Ran 84 tests in 19.830s FAILED (failures=1, skipped=2) ================================================================================ ==================== Test output for //tensorflow/python/kernel_tests/nn_ops:conv_ops_test_cpu (shard 3 of 4): 2023-05-20 05:32:18.696859: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. Running tests under Python 3.10.11: /usr/local/bin/python3 [ RUN ] Conv2DTest.testConv2D1x1Padding INFO:tensorflow:Running testConv2D1x1Padding in GRAPH mode. I0520 05:32:21.720572 281473723757440 test_util.py:1494] Running testConv2D1x1Padding in GRAPH mode. WARNING:tensorflow:From /usr/lib/python3.10/contextlib.py:103: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `self.session()` or `self.cached_session()` instead. W0520 05:32:21.721025 281473723757440 deprecation.py:364] From /usr/lib/python3.10/contextlib.py:103: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `self.session()` or `self.cached_session()` instead. 2023-05-20 05:32:21.763581: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:375] MLIR V1 optimization pass is not enabled WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py:154: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. W0520 05:32:21.898422 281473723757440 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py:154: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. 2023-05-20 05:32:21.898878: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D1x1Padding): 0.32s I0520 05:32:22.038079 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2D1x1Padding): 0.32s INFO:tensorflow:Running testConv2D1x1Padding in EAGER mode. I0520 05:32:22.039626 281473723757440 test_util.py:1503] Running testConv2D1x1Padding in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D1x1Padding): 0.1s I0520 05:32:22.144034 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2D1x1Padding): 0.1s [ OK ] Conv2DTest.testConv2D1x1Padding [ RUN ] Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput INFO:tensorflow:Running testConv2D2x2Depth1Padding0x0BackpropInput in GRAPH mode. I0520 05:32:22.145349 281473723757440 test_util.py:1494] Running testConv2D2x2Depth1Padding0x0BackpropInput in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput): 0.09s I0520 05:32:22.233020 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput): 0.09s INFO:tensorflow:Running testConv2D2x2Depth1Padding0x0BackpropInput in EAGER mode. I0520 05:32:22.234234 281473723757440 test_util.py:1503] Running testConv2D2x2Depth1Padding0x0BackpropInput in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput): 0.04s I0520 05:32:22.276963 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput): 0.04s [ OK ] Conv2DTest.testConv2D2x2Depth1Padding0x0BackpropInput [ RUN ] Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput INFO:tensorflow:Running testConv2D2x2Depth1Padding2x2BackpropInput in GRAPH mode. I0520 05:32:22.278221 281473723757440 test_util.py:1494] Running testConv2D2x2Depth1Padding2x2BackpropInput in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput): 0.04s I0520 05:32:22.319873 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput): 0.04s INFO:tensorflow:Running testConv2D2x2Depth1Padding2x2BackpropInput in EAGER mode. I0520 05:32:22.320867 281473723757440 test_util.py:1503] Running testConv2D2x2Depth1Padding2x2BackpropInput in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput): 0.02s I0520 05:32:22.343099 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput): 0.02s [ OK ] Conv2DTest.testConv2D2x2Depth1Padding2x2BackpropInput [ RUN ] Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput INFO:tensorflow:Running testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput in GRAPH mode. I0520 05:32:22.344238 281473723757440 test_util.py:1494] Running testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput): 0.04s I0520 05:32:22.387561 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput): 0.04s INFO:tensorflow:Running testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput in EAGER mode. I0520 05:32:22.388587 281473723757440 test_util.py:1503] Running testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput): 0.02s I0520 05:32:22.413355 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput): 0.02s [ OK ] Conv2DTest.testConv2D2x2Depth1Padding_5_0_2_2_BackpropInput [ RUN ] Conv2DTest.testConv2D2x2Depth1ValidBackpropInputDilation1x2 INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth1ValidBackpropInputDilation1x2): 0.1s I0520 05:32:22.511162 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth1ValidBackpropInputDilation1x2): 0.1s [ OK ] Conv2DTest.testConv2D2x2Depth1ValidBackpropInputDilation1x2 [ RUN ] Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2 INFO:tensorflow:Running testConv2D2x2Depth3ValidBackpropFilterStride1x2 in GRAPH mode. I0520 05:32:22.512505 281473723757440 test_util.py:1494] Running testConv2D2x2Depth3ValidBackpropFilterStride1x2 in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2): 0.06s I0520 05:32:22.567732 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2): 0.06s INFO:tensorflow:Running testConv2D2x2Depth3ValidBackpropFilterStride1x2 in EAGER mode. I0520 05:32:22.568809 281473723757440 test_util.py:1503] Running testConv2D2x2Depth3ValidBackpropFilterStride1x2 in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2): 0.02s I0520 05:32:22.591285 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2): 0.02s [ OK ] Conv2DTest.testConv2D2x2Depth3ValidBackpropFilterStride1x2 [ RUN ] Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2 INFO:tensorflow:Running testConv2D2x2Depth3ValidBackpropInputStride1x2 in GRAPH mode. I0520 05:32:22.592279 281473723757440 test_util.py:1494] Running testConv2D2x2Depth3ValidBackpropInputStride1x2 in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2): 0.02s I0520 05:32:22.616095 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2): 0.02s INFO:tensorflow:Running testConv2D2x2Depth3ValidBackpropInputStride1x2 in EAGER mode. I0520 05:32:22.617105 281473723757440 test_util.py:1503] Running testConv2D2x2Depth3ValidBackpropInputStride1x2 in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2): 0.01s I0520 05:32:22.630296 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2): 0.01s [ OK ] Conv2DTest.testConv2D2x2Depth3ValidBackpropInputStride1x2 [ RUN ] Conv2DTest.testConv2D2x2FilterStride1x2 INFO:tensorflow:Running testConv2D2x2FilterStride1x2 in GRAPH mode. I0520 05:32:22.631253 281473723757440 test_util.py:1494] Running testConv2D2x2FilterStride1x2 in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2FilterStride1x2): 0.06s I0520 05:32:22.690765 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2FilterStride1x2): 0.06s INFO:tensorflow:Running testConv2D2x2FilterStride1x2 in EAGER mode. I0520 05:32:22.691888 281473723757440 test_util.py:1503] Running testConv2D2x2FilterStride1x2 in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2D2x2FilterStride1x2): 0.02s I0520 05:32:22.714030 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2D2x2FilterStride1x2): 0.02s [ OK ] Conv2DTest.testConv2D2x2FilterStride1x2 [ RUN ] Conv2DTest.testConv2DBackpropFilterWithEmptyInput INFO:tensorflow:Running testConv2DBackpropFilterWithEmptyInput in GRAPH mode. I0520 05:32:22.715084 281473723757440 test_util.py:1494] Running testConv2DBackpropFilterWithEmptyInput in GRAPH mode. 2023-05-20 05:32:22.725945: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2DBackpropFilter" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 1 } } } } inputs { dtype: DT_INT32 shape { dim { size: 4 } } value { dtype: DT_INT32 tensor_shape { dim { size: 4 } } tensor_content: "\002\000\000\000\002\000\000\000\001\000\000\000\001\000\000\000" } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 1 } dim { size: 2 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 1 } dim { size: 2 } dim { size: 1 } } } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: 2 } dim { size: 2 } dim { size: 1 } dim { size: 1 } } } 2023-05-20 05:32:22.743919: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2DBackpropFilter" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 1 } } } } inputs { dtype: DT_INT32 shape { dim { size: 4 } } value { dtype: DT_INT32 tensor_shape { dim { size: 4 } } tensor_content: "\002\000\000\000\002\000\000\000\001\000\000\000\001\000\000\000" } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 1 } dim { size: 2 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 1 } dim { size: 2 } dim { size: 1 } } } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: 2 } dim { size: 2 } dim { size: 1 } dim { size: 1 } } } INFO:tensorflow:time(__main__.Conv2DTest.testConv2DBackpropFilterWithEmptyInput): 0.04s I0520 05:32:22.752804 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2DBackpropFilterWithEmptyInput): 0.04s INFO:tensorflow:Running testConv2DBackpropFilterWithEmptyInput in EAGER mode. I0520 05:32:22.753842 281473723757440 test_util.py:1503] Running testConv2DBackpropFilterWithEmptyInput in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DBackpropFilterWithEmptyInput): 0.01s I0520 05:32:22.764739 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2DBackpropFilterWithEmptyInput): 0.01s [ OK ] Conv2DTest.testConv2DBackpropFilterWithEmptyInput [ RUN ] Conv2DTest.testConv2DEmptyBackpropFilter INFO:tensorflow:Running testConv2DEmptyBackpropFilter in GRAPH mode. I0520 05:32:22.765752 281473723757440 test_util.py:1494] Running testConv2DEmptyBackpropFilter in GRAPH mode. 2023-05-20 05:32:22.775936: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2DBackpropFilter" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 2 } dim { size: 3 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 2 } dim { size: 3 } dim { size: 1 } } tensor_content: "\000\000\200?\000\000\000@\000\000@@\000\000\200@\000\000\240@\000\000\300@" } } inputs { dtype: DT_INT32 shape { dim { size: 4 } } value { dtype: DT_INT32 tensor_shape { dim { size: 4 } } tensor_content: "\002\000\000\000\002\000\000\000\001\000\000\000\000\000\000\000" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1 } dim { size: 2 } dim { } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 1 } dim { size: 2 } dim { } } } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: 2 } dim { size: 2 } dim { size: 1 } dim { } } } 2023-05-20 05:32:22.793144: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2DBackpropFilter" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 2 } dim { size: 3 } dim { size: 1 } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 2 } dim { size: 3 } dim { size: 1 } } tensor_content: "\000\000\200?\000\000\000@\000\000@@\000\000\200@\000\000\240@\000\000\300@" } } inputs { dtype: DT_INT32 shape { dim { size: 4 } } value { dtype: DT_INT32 tensor_shape { dim { size: 4 } } tensor_content: "\002\000\000\000\002\000\000\000\001\000\000\000\000\000\000\000" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1 } dim { size: 2 } dim { } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 1 } dim { size: 2 } dim { } } } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: 2 } dim { size: 2 } dim { size: 1 } dim { } } } INFO:tensorflow:time(__main__.Conv2DTest.testConv2DEmptyBackpropFilter): 0.04s I0520 05:32:22.809431 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2DEmptyBackpropFilter): 0.04s INFO:tensorflow:Running testConv2DEmptyBackpropFilter in EAGER mode. I0520 05:32:22.810501 281473723757440 test_util.py:1503] Running testConv2DEmptyBackpropFilter in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DEmptyBackpropFilter): 0.01s I0520 05:32:22.821816 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2DEmptyBackpropFilter): 0.01s [ OK ] Conv2DTest.testConv2DEmptyBackpropFilter [ RUN ] Conv2DTest.testConv2DEmptyDilation INFO:tensorflow:Running testConv2DEmptyDilation in GRAPH mode. I0520 05:32:22.822780 281473723757440 test_util.py:1494] Running testConv2DEmptyDilation in GRAPH mode. 2023-05-20 05:32:22.857737: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2D" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 1 } dim { size: 3 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } tensor_content: "\000\000\200?\000\000\000@\000\000@@\000\000\200@\000\000\240@\000\000\300@\000\000\340@\000\000\000A\000\000\020A" } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { } dim { size: 1 } dim { size: 3 } dim { size: 3 } } } 2023-05-20 05:32:22.872567: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2D" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 2 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } tensor_content: "\000\000\200?\000\000\000@\000\000@@\000\000\200@\000\000\240@\000\000\300@\000\000\340@\000\000\000A\000\000\020A" } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } } 2023-05-20 05:32:22.872880: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:693] Error in PredictCost() for the op: op: "Conv2D" attr { key: "T" value { type: DT_FLOAT } } attr { key: "data_format" value { s: "NHWC" } } attr { key: "dilations" value { list { i: 1 i: 2 i: 1 i: 1 } } } attr { key: "explicit_paddings" value { list { } } } attr { key: "padding" value { s: "VALID" } } attr { key: "strides" value { list { i: 1 i: 1 i: 1 i: 1 } } } attr { key: "use_cudnn_on_gpu" value { b: true } } inputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } value { dtype: DT_FLOAT tensor_shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } value { dtype: DT_FLOAT tensor_shape { dim { size: 1 } dim { size: 1 } dim { size: 3 } dim { size: 3 } } tensor_content: "\000\000\200?\000\000\000@\000\000@@\000\000\200@\000\000\240@\000\000\300@\000\000\340@\000\000\000A\000\000\020A" } } device { type: "CPU" model: "0" frequency: 200 num_cores: 56 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { } dim { size: 2 } dim { size: 3 } dim { size: 3 } } } INFO:tensorflow:time(__main__.Conv2DTest.testConv2DEmptyDilation): 0.06s I0520 05:32:22.883702 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2DEmptyDilation): 0.06s INFO:tensorflow:Running testConv2DEmptyDilation in EAGER mode. I0520 05:32:22.884899 281473723757440 test_util.py:1503] Running testConv2DEmptyDilation in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DEmptyDilation): 0.02s I0520 05:32:22.909558 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2DEmptyDilation): 0.02s [ OK ] Conv2DTest.testConv2DEmptyDilation [ RUN ] Conv2DTest.testConv2DGroupConvFwd INFO:tensorflow:Running testConv2DGroupConvFwd in GRAPH mode. I0520 05:32:22.910722 281473723757440 test_util.py:1494] Running testConv2DGroupConvFwd in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DGroupConvFwd): 0.82s I0520 05:32:23.733064 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2DGroupConvFwd): 0.82s [ FAILED ] Conv2DTest.testConv2DGroupConvFwd [ RUN ] Conv2DTest.testConv2DKernelSizeMatchesInputSizeBackpropFilterDilation2x2 INFO:tensorflow:time(__main__.Conv2DTest.testConv2DKernelSizeMatchesInputSizeBackpropFilterDilation2x2): 0.07s I0520 05:32:23.805361 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2DKernelSizeMatchesInputSizeBackpropFilterDilation2x2): 0.07s [ OK ] Conv2DTest.testConv2DKernelSizeMatchesInputSizeBackpropFilterDilation2x2 [ RUN ] Conv2DTest.testConv2DKernelSmallerThanStrideSame INFO:tensorflow:Running testConv2DKernelSmallerThanStrideSame in GRAPH mode. I0520 05:32:23.806795 281473723757440 test_util.py:1494] Running testConv2DKernelSmallerThanStrideSame in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DKernelSmallerThanStrideSame): 0.18s I0520 05:32:23.990515 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2DKernelSmallerThanStrideSame): 0.18s INFO:tensorflow:Running testConv2DKernelSmallerThanStrideSame in EAGER mode. I0520 05:32:23.992268 281473723757440 test_util.py:1503] Running testConv2DKernelSmallerThanStrideSame in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DKernelSmallerThanStrideSame): 0.07s I0520 05:32:24.059401 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2DKernelSmallerThanStrideSame): 0.07s [ OK ] Conv2DTest.testConv2DKernelSmallerThanStrideSame [ RUN ] Conv2DTest.testConv2DOnlyTopRightPadding INFO:tensorflow:Running testConv2DOnlyTopRightPadding in GRAPH mode. I0520 05:32:24.060599 281473723757440 test_util.py:1494] Running testConv2DOnlyTopRightPadding in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DOnlyTopRightPadding): 0.17s I0520 05:32:24.231262 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2DOnlyTopRightPadding): 0.17s INFO:tensorflow:Running testConv2DOnlyTopRightPadding in EAGER mode. I0520 05:32:24.233046 281473723757440 test_util.py:1503] Running testConv2DOnlyTopRightPadding in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConv2DOnlyTopRightPadding): 0.14s I0520 05:32:24.373329 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConv2DOnlyTopRightPadding): 0.14s [ OK ] Conv2DTest.testConv2DOnlyTopRightPadding [ RUN ] Conv2DTest.testConvolutionClass2DExpandedBatch INFO:tensorflow:Running testConvolutionClass2DExpandedBatch in GRAPH mode. I0520 05:32:24.374928 281473723757440 test_util.py:1494] Running testConvolutionClass2DExpandedBatch in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testConvolutionClass2DExpandedBatch): 0.14s I0520 05:32:24.519641 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConvolutionClass2DExpandedBatch): 0.14s INFO:tensorflow:Running testConvolutionClass2DExpandedBatch in EAGER mode. I0520 05:32:24.520667 281473723757440 test_util.py:1503] Running testConvolutionClass2DExpandedBatch in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testConvolutionClass2DExpandedBatch): 0.02s I0520 05:32:24.544876 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testConvolutionClass2DExpandedBatch): 0.02s [ OK ] Conv2DTest.testConvolutionClass2DExpandedBatch [ RUN ] Conv2DTest.testFilterGradient1x1PaddingStrideOne WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py:1953: compute_gradient (from tensorflow.python.ops.gradient_checker) is deprecated and will be removed in a future version. Instructions for updating: Use tf.test.compute_gradient in 2.0, which has better support for functions. Note that the two versions have different usage, so code change is needed. W0520 05:32:24.549824 281473723757440 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py:1953: compute_gradient (from tensorflow.python.ops.gradient_checker) is deprecated and will be removed in a future version. Instructions for updating: Use tf.test.compute_gradient in 2.0, which has better support for functions. Note that the two versions have different usage, so code change is needed. INFO:tensorflow:time(__main__.Conv2DTest.testFilterGradient1x1PaddingStrideOne): 0.75s I0520 05:32:25.291104 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testFilterGradient1x1PaddingStrideOne): 0.75s [ OK ] Conv2DTest.testFilterGradient1x1PaddingStrideOne [ RUN ] Conv2DTest.testFilterGradientGroupConv INFO:tensorflow:time(__main__.Conv2DTest.testFilterGradientGroupConv): 0.0s I0520 05:32:25.292582 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testFilterGradientGroupConv): 0.0s [ SKIPPED ] Conv2DTest.testFilterGradientGroupConv [ RUN ] Conv2DTest.testFilterGradientSamePaddingStrideThree INFO:tensorflow:time(__main__.Conv2DTest.testFilterGradientSamePaddingStrideThree): 1.26s I0520 05:32:26.548384 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testFilterGradientSamePaddingStrideThree): 1.26s [ OK ] Conv2DTest.testFilterGradientSamePaddingStrideThree [ RUN ] Conv2DTest.testFilterGradientValidPaddingStrideTwo INFO:tensorflow:time(__main__.Conv2DTest.testFilterGradientValidPaddingStrideTwo): 1.55s I0520 05:32:28.103413 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testFilterGradientValidPaddingStrideTwo): 1.55s [ OK ] Conv2DTest.testFilterGradientValidPaddingStrideTwo [ RUN ] Conv2DTest.testInceptionBackFilter_11 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:28.105276 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], [1, 1], 'SAME') I0520 05:32:28.106418 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_11): 0.06s I0520 05:32:28.160717 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_11): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:28.162101 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], [1, 1], 'SAME') I0520 05:32:28.162307 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_11): 0.02s I0520 05:32:28.184055 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_11): 0.02s [ OK ] Conv2DTest.testInceptionBackFilter_11 [ RUN ] Conv2DTest.testInceptionBackFilter_15 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:28.185346 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], [2, 2], 'VALID') I0520 05:32:28.186566 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], [2, 2], 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_15): 0.05s I0520 05:32:28.234256 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_15): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:28.235702 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], [2, 2], 'VALID') I0520 05:32:28.235958 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], [2, 2], 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_15): 0.02s I0520 05:32:28.256646 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_15): 0.02s [ OK ] Conv2DTest.testInceptionBackFilter_15 [ RUN ] Conv2DTest.testInceptionBackFilter_19 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:28.257923 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], [1, 1], 'SAME') I0520 05:32:28.259023 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_19): 0.06s I0520 05:32:28.313768 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_19): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:28.315107 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], [1, 1], 'SAME') I0520 05:32:28.315302 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_19): 0.02s I0520 05:32:28.337816 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_19): 0.02s [ OK ] Conv2DTest.testInceptionBackFilter_19 [ RUN ] Conv2DTest.testInceptionBackFilter_22 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:28.338864 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], [1, 1], 'SAME') I0520 05:32:28.339776 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_22): 0.05s I0520 05:32:28.390699 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_22): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:28.391815 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], [1, 1], 'SAME') I0520 05:32:28.392015 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_22): 0.02s I0520 05:32:28.414195 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_22): 0.02s [ OK ] Conv2DTest.testInceptionBackFilter_22 [ RUN ] Conv2DTest.testInceptionBackFilter_26 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:28.415502 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], [1, 1], 'SAME') I0520 05:32:28.416710 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_26): 0.07s I0520 05:32:28.487863 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_26): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:28.489261 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], [1, 1], 'SAME') I0520 05:32:28.489491 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_26): 0.03s I0520 05:32:28.515143 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_26): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_26 [ RUN ] Conv2DTest.testInceptionBackFilter_3 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:28.516427 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], [1, 1], 'SAME') I0520 05:32:28.517572 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_3): 0.06s I0520 05:32:28.573946 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_3): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:28.575488 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], [1, 1], 'SAME') I0520 05:32:28.575789 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_3): 0.02s I0520 05:32:28.598637 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_3): 0.02s [ OK ] Conv2DTest.testInceptionBackFilter_3 [ RUN ] Conv2DTest.testInceptionBackFilter_33 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:28.599987 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], [1, 1], 'SAME') I0520 05:32:28.601176 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_33): 0.05s I0520 05:32:28.650765 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_33): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:28.652174 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], [1, 1], 'SAME') I0520 05:32:28.652432 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_33): 0.02s I0520 05:32:28.672891 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_33): 0.02s [ OK ] Conv2DTest.testInceptionBackFilter_33 [ RUN ] Conv2DTest.testInceptionBackFilter_37 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:28.674134 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], [1, 1], 'SAME') I0520 05:32:28.675207 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_37): 0.05s I0520 05:32:28.722959 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_37): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:28.724308 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], [1, 1], 'SAME') I0520 05:32:28.724506 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_37): 0.02s I0520 05:32:28.745562 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_37): 0.02s [ OK ] Conv2DTest.testInceptionBackFilter_37 [ RUN ] Conv2DTest.testInceptionBackFilter_40 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:28.746873 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], [1, 1], 'SAME') I0520 05:32:28.748024 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_40): 0.08s I0520 05:32:28.822977 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_40): 0.08s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:28.824499 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], [1, 1], 'SAME') I0520 05:32:28.824764 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_40): 0.03s I0520 05:32:28.849673 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_40): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_40 [ RUN ] Conv2DTest.testInceptionBackFilter_44 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:28.851004 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], [1, 1], 'SAME') I0520 05:32:28.852176 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_44): 0.08s I0520 05:32:28.930321 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_44): 0.08s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:28.931691 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], [1, 1], 'SAME') I0520 05:32:28.931894 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_44): 0.03s I0520 05:32:28.958765 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_44): 0.03s [ OK ] Conv2DTest.testInceptionBackFilter_44 [ RUN ] Conv2DTest.testInceptionBackFilter_48 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:28.960039 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], [1, 1], 'SAME') I0520 05:32:28.961241 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_48): 0.13s I0520 05:32:29.085084 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_48): 0.13s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:29.086714 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], [1, 1], 'SAME') I0520 05:32:29.087010 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_48): 0.04s I0520 05:32:29.128278 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_48): 0.04s [ OK ] Conv2DTest.testInceptionBackFilter_48 [ RUN ] Conv2DTest.testInceptionBackFilter_51 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:29.129565 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], [1, 1], 'SAME') I0520 05:32:29.130757 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_51): 0.11s I0520 05:32:29.242650 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_51): 0.11s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:29.244057 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], [1, 1], 'SAME') I0520 05:32:29.244245 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_51): 0.02s I0520 05:32:29.268992 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_51): 0.02s [ OK ] Conv2DTest.testInceptionBackFilter_51 [ RUN ] Conv2DTest.testInceptionBackFilter_6 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:29.270292 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], [1, 1], 'SAME') I0520 05:32:29.271396 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_6): 0.06s I0520 05:32:29.329442 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_6): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:29.330813 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], [1, 1], 'SAME') I0520 05:32:29.331034 281473723757440 conv_ops_test.py:3257] Testing InceptionBackFilter ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_6): 0.02s I0520 05:32:29.352087 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_6): 0.02s [ OK ] Conv2DTest.testInceptionBackFilter_6 [ RUN ] Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:29.353384 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Skipping InceptionBackFilter ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], [1, 1], 'SAME') I0520 05:32:29.354542 281473723757440 conv_ops_test.py:3254] Skipping InceptionBackFilter ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused): 0.0s I0520 05:32:29.354815 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused): 0.0s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:29.355400 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Skipping InceptionBackFilter ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], [1, 1], 'SAME') I0520 05:32:29.355624 281473723757440 conv_ops_test.py:3254] Skipping InceptionBackFilter ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], [1, 1], 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused): 0.0s I0520 05:32:29.355886 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused): 0.0s [ OK ] Conv2DTest.testInceptionBackFilter_No_Winograd_Nonfused [ RUN ] Conv2DTest.testInceptionBackInput_11 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:29.356561 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], 1, 'SAME') I0520 05:32:29.357279 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_11): 0.05s I0520 05:32:29.405425 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_11): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:29.406858 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], 1, 'SAME') I0520 05:32:29.407100 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 176], [1, 1, 176, 32], [4, 8, 8, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_11): 0.04s I0520 05:32:29.446004 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_11): 0.04s [ OK ] Conv2DTest.testInceptionBackInput_11 [ RUN ] Conv2DTest.testInceptionBackInput_15 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:29.447294 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], 2, 'VALID') I0520 05:32:29.448418 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], 2, 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_15): 0.05s I0520 05:32:29.494802 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_15): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:29.496261 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], 2, 'VALID') I0520 05:32:29.496484 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [3, 3, 12, 32], [4, 8, 8, 32], 2, 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_15): 0.06s I0520 05:32:29.559782 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_15): 0.06s [ OK ] Conv2DTest.testInceptionBackInput_15 [ RUN ] Conv2DTest.testInceptionBackInput_19 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:29.561097 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], 1, 'SAME') I0520 05:32:29.562250 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_19): 0.06s I0520 05:32:29.624788 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_19): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:29.626299 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], 1, 'SAME') I0520 05:32:29.626547 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 19], [1, 3, 19, 25], [4, 17, 17, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_19): 0.02s I0520 05:32:29.644361 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_19): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_19 [ RUN ] Conv2DTest.testInceptionBackInput_22 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:29.645622 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], 1, 'SAME') I0520 05:32:29.646798 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_22): 0.05s I0520 05:32:29.695097 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_22): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:29.696575 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], 1, 'SAME') I0520 05:32:29.696804 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 22], [3, 1, 22, 22], [4, 17, 17, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_22): 0.02s I0520 05:32:29.714620 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_22): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_22 [ RUN ] Conv2DTest.testInceptionBackInput_26 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:29.715918 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], 1, 'SAME') I0520 05:32:29.717044 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_26): 0.07s I0520 05:32:29.782573 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_26): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:29.783929 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], 1, 'SAME') I0520 05:32:29.784129 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 115], [1, 1, 115, 12], [4, 17, 17, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_26): 0.02s I0520 05:32:29.803345 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_26): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_26 [ RUN ] Conv2DTest.testInceptionBackInput_3 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:29.804624 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], 1, 'SAME') I0520 05:32:29.805788 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_3): 0.07s I0520 05:32:29.874256 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_3): 0.07s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:29.875613 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], 1, 'SAME') I0520 05:32:29.875812 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 19], [4, 8, 8, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_3): 0.03s I0520 05:32:29.903890 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_3): 0.03s [ OK ] Conv2DTest.testInceptionBackInput_3 [ RUN ] Conv2DTest.testInceptionBackInput_33 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:29.905120 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], 1, 'SAME') I0520 05:32:29.906178 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_33): 0.05s I0520 05:32:29.954121 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_33): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:29.955462 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], 1, 'SAME') I0520 05:32:29.955667 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [3, 1, 12, 19], [4, 17, 17, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_33): 0.02s I0520 05:32:29.971943 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_33): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_33 [ RUN ] Conv2DTest.testInceptionBackInput_37 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:29.973177 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], 1, 'SAME') I0520 05:32:29.974235 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_37): 0.05s I0520 05:32:30.021774 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_37): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:30.023227 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], 1, 'SAME') I0520 05:32:30.023446 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 12], [1, 3, 12, 12], [4, 17, 17, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_37): 0.02s I0520 05:32:30.039635 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_37): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_37 [ RUN ] Conv2DTest.testInceptionBackInput_40 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:30.040924 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], 1, 'SAME') I0520 05:32:30.042102 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_40): 0.05s I0520 05:32:30.094619 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_40): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:30.096138 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], 1, 'SAME') I0520 05:32:30.096407 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 17, 17, 76], [1, 1, 76, 32], [4, 17, 17, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_40): 0.02s I0520 05:32:30.116098 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_40): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_40 [ RUN ] Conv2DTest.testInceptionBackInput_44 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:30.117410 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], 1, 'SAME') I0520 05:32:30.118495 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_44): 0.06s I0520 05:32:30.181489 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_44): 0.06s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:30.182900 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], 1, 'SAME') I0520 05:32:30.183104 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 28], [1, 1, 28, 6], [4, 35, 35, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_44): 0.06s I0520 05:32:30.241385 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_44): 0.06s [ OK ] Conv2DTest.testInceptionBackInput_44 [ RUN ] Conv2DTest.testInceptionBackInput_48 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:30.242703 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], 1, 'SAME') I0520 05:32:30.243903 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_48): 0.09s I0520 05:32:30.333220 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_48): 0.09s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:30.334731 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], 1, 'SAME') I0520 05:32:30.334970 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 9], [3, 3, 9, 9], [4, 35, 35, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_48): 0.03s I0520 05:32:30.362386 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_48): 0.03s [ OK ] Conv2DTest.testInceptionBackInput_48 [ RUN ] Conv2DTest.testInceptionBackInput_51 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:30.363686 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], 1, 'SAME') I0520 05:32:30.364835 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_51): 0.05s I0520 05:32:30.414223 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_51): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:30.415637 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], 1, 'SAME') I0520 05:32:30.415851 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 35, 35, 19], [1, 1, 19, 4], [4, 35, 35, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_51): 0.02s I0520 05:32:30.433787 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_51): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_51 [ RUN ] Conv2DTest.testInceptionBackInput_6 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:30.435074 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], 1, 'SAME') I0520 05:32:30.436200 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_6): 0.05s I0520 05:32:30.484498 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_6): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:30.485944 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], 1, 'SAME') I0520 05:32:30.486196 281473723757440 conv_ops_test.py:3241] Testing InceptionBackInput ([4, 8, 8, 204], [1, 1, 204, 44], [4, 8, 8, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_6): 0.02s I0520 05:32:30.502997 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_6): 0.02s [ OK ] Conv2DTest.testInceptionBackInput_6 [ RUN ] Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:30.504272 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Skipping InceptionBackInput ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], 1, 'SAME') I0520 05:32:30.505451 281473723757440 conv_ops_test.py:3238] Skipping InceptionBackInput ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused): 0.0s I0520 05:32:30.505740 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused): 0.0s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:30.506388 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Skipping InceptionBackInput ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], 1, 'SAME') I0520 05:32:30.506638 281473723757440 conv_ops_test.py:3238] Skipping InceptionBackInput ([1, 400, 400, 1], [1, 1, 1, 256], [1, 400, 400, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused): 0.0s I0520 05:32:30.506901 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused): 0.0s [ OK ] Conv2DTest.testInceptionBackInput_No_Winograd_Nonfused [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_11 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:30.507547 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') I0520 05:32:30.508305 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_11): 0.16s I0520 05:32:30.667732 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_11): 0.16s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:30.669322 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') I0520 05:32:30.669549 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_11): 0.08s I0520 05:32:30.751373 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_11): 0.08s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_11 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_15 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:30.752905 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_15): 0.0s I0520 05:32:30.754095 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_15): 0.0s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:30.754782 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_15): 0.0s I0520 05:32:30.755044 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_15): 0.0s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_15 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_19 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:30.755692 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') I0520 05:32:30.756485 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_19): 0.09s I0520 05:32:30.845689 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_19): 0.09s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:30.847481 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') I0520 05:32:30.847786 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_19): 0.05s I0520 05:32:30.896593 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_19): 0.05s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_19 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_22 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:30.898284 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') I0520 05:32:30.899456 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_22): 0.11s I0520 05:32:31.008440 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_22): 0.11s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:31.010045 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') I0520 05:32:31.010285 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_22): 0.04s I0520 05:32:31.046483 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_22): 0.04s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_22 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_26 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:31.048062 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') I0520 05:32:31.049165 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_26): 0.2s I0520 05:32:31.245155 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_26): 0.2s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:31.246848 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') I0520 05:32:31.247117 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_26): 0.04s I0520 05:32:31.291163 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_26): 0.04s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_26 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_3 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:31.292747 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') I0520 05:32:31.293835 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_3): 0.1s I0520 05:32:31.396050 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_3): 0.1s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:31.397730 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') I0520 05:32:31.398016 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_3): 0.05s I0520 05:32:31.443088 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_3): 0.05s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_3 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_33 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:31.444554 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') I0520 05:32:31.445675 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_33): 0.11s I0520 05:32:31.553407 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_33): 0.11s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:31.555024 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') I0520 05:32:31.555255 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_33): 0.05s I0520 05:32:31.605016 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_33): 0.05s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_33 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_37 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:31.606605 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') I0520 05:32:31.607796 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_37): 0.09s I0520 05:32:31.699012 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_37): 0.09s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:31.700610 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') I0520 05:32:31.700859 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_37): 0.03s I0520 05:32:31.734760 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_37): 0.03s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_37 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_40 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:31.736366 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') I0520 05:32:31.737502 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_40): 0.13s I0520 05:32:31.864202 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_40): 0.13s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:31.865875 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') I0520 05:32:31.866098 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_40): 0.07s I0520 05:32:31.935312 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_40): 0.07s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_40 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_44 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:31.936880 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') I0520 05:32:31.938066 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_44): 0.11s I0520 05:32:32.046464 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_44): 0.11s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:32.048026 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') I0520 05:32:32.048255 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_44): 0.04s I0520 05:32:32.092899 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_44): 0.04s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_44 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_48 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:32.094405 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') I0520 05:32:32.095547 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_48): 0.1s I0520 05:32:32.191109 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_48): 0.1s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:32.192925 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') I0520 05:32:32.193176 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_48): 0.06s I0520 05:32:32.255855 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_48): 0.06s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_48 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_51 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:32.257513 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') I0520 05:32:32.258663 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_51): 0.17s I0520 05:32:32.430053 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_51): 0.17s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:32.431848 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') I0520 05:32:32.432064 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_51): 0.07s I0520 05:32:32.502037 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_51): 0.07s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_51 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_6 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:32.503651 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') I0520 05:32:32.504917 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_6): 0.11s I0520 05:32:32.609412 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_6): 0.11s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:32.611209 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') I0520 05:32:32.611503 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_6): 0.04s I0520 05:32:32.653066 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_6): 0.04s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_6 [ RUN ] Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:32.654659 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd with dilations ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') I0520 05:32:32.655968 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused): 1.76s I0520 05:32:34.419339 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused): 1.76s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:34.421069 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd with dilations ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') I0520 05:32:34.421360 281473723757440 conv_ops_test.py:3219] Testing InceptionFwd with dilations ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused): 1.62s I0520 05:32:36.044703 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused): 1.62s [ OK ] Conv2DTest.testInceptionFwdDilatedConv_No_Winograd_Nonfused [ RUN ] Conv2DTest.testInceptionFwd_11 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:36.046359 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') I0520 05:32:36.047626 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_11): 0.04s I0520 05:32:36.083227 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_11): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:36.084554 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') I0520 05:32:36.084783 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 176], [1, 1, 176, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_11): 0.03s I0520 05:32:36.117927 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_11): 0.03s [ OK ] Conv2DTest.testInceptionFwd_11 [ RUN ] Conv2DTest.testInceptionFwd_15 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:36.119156 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [3, 3, 12, 32], 2, 'VALID') I0520 05:32:36.120196 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [3, 3, 12, 32], 2, 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_15): 0.03s I0520 05:32:36.153174 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_15): 0.03s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:36.154511 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [3, 3, 12, 32], 2, 'VALID') I0520 05:32:36.154751 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [3, 3, 12, 32], 2, 'VALID') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_15): 0.01s I0520 05:32:36.169459 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_15): 0.01s [ OK ] Conv2DTest.testInceptionFwd_15 [ RUN ] Conv2DTest.testInceptionFwd_19 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:36.170711 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') I0520 05:32:36.171725 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_19): 0.04s I0520 05:32:36.210784 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_19): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:36.212057 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') I0520 05:32:36.212276 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 19], [1, 3, 19, 25], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_19): 0.01s I0520 05:32:36.223885 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_19): 0.01s [ OK ] Conv2DTest.testInceptionFwd_19 [ RUN ] Conv2DTest.testInceptionFwd_22 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:36.225093 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') I0520 05:32:36.226148 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_22): 0.04s I0520 05:32:36.267552 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_22): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:36.268832 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') I0520 05:32:36.269044 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 22], [3, 1, 22, 22], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_22): 0.01s I0520 05:32:36.280569 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_22): 0.01s [ OK ] Conv2DTest.testInceptionFwd_22 [ RUN ] Conv2DTest.testInceptionFwd_26 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:36.281779 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') I0520 05:32:36.282789 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_26): 0.04s I0520 05:32:36.324077 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_26): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:36.325384 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') I0520 05:32:36.325602 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 115], [1, 1, 115, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_26): 0.02s I0520 05:32:36.346397 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_26): 0.02s [ OK ] Conv2DTest.testInceptionFwd_26 [ RUN ] Conv2DTest.testInceptionFwd_3 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:36.347630 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') I0520 05:32:36.348684 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_3): 0.05s I0520 05:32:36.398750 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_3): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:36.399996 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') I0520 05:32:36.400203 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_3): 0.01s I0520 05:32:36.411782 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_3): 0.01s [ OK ] Conv2DTest.testInceptionFwd_3 [ RUN ] Conv2DTest.testInceptionFwd_33 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:36.412967 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') I0520 05:32:36.413949 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_33): 0.04s I0520 05:32:36.456300 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_33): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:36.457569 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') I0520 05:32:36.457781 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [3, 1, 12, 19], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_33): 0.02s I0520 05:32:36.473504 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_33): 0.02s [ OK ] Conv2DTest.testInceptionFwd_33 [ RUN ] Conv2DTest.testInceptionFwd_37 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:36.474675 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') I0520 05:32:36.475679 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_37): 0.03s I0520 05:32:36.507552 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_37): 0.03s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:36.508793 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') I0520 05:32:36.509000 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 12], [1, 3, 12, 12], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_37): 0.02s I0520 05:32:36.530215 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_37): 0.02s [ OK ] Conv2DTest.testInceptionFwd_37 [ RUN ] Conv2DTest.testInceptionFwd_40 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:36.531468 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') I0520 05:32:36.532592 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_40): 0.04s I0520 05:32:36.572429 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_40): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:36.573793 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') I0520 05:32:36.574007 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 17, 17, 76], [1, 1, 76, 32], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_40): 0.02s I0520 05:32:36.591354 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_40): 0.02s [ OK ] Conv2DTest.testInceptionFwd_40 [ RUN ] Conv2DTest.testInceptionFwd_44 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:36.592598 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') I0520 05:32:36.593697 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_44): 0.05s I0520 05:32:36.638510 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_44): 0.05s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:36.639894 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') I0520 05:32:36.640141 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 28], [1, 1, 28, 6], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_44): 0.03s I0520 05:32:36.669119 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_44): 0.03s [ OK ] Conv2DTest.testInceptionFwd_44 [ RUN ] Conv2DTest.testInceptionFwd_48 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:36.670356 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') I0520 05:32:36.671425 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_48): 0.04s I0520 05:32:36.706703 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_48): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:36.707996 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') I0520 05:32:36.708216 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 9], [3, 3, 9, 9], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_48): 0.02s I0520 05:32:36.723964 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_48): 0.02s [ OK ] Conv2DTest.testInceptionFwd_48 [ RUN ] Conv2DTest.testInceptionFwd_51 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:36.725211 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') I0520 05:32:36.726418 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_51): 0.04s I0520 05:32:36.767409 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_51): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:36.768668 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') I0520 05:32:36.768875 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 35, 35, 19], [1, 1, 19, 4], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_51): 0.02s I0520 05:32:36.790712 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_51): 0.02s [ OK ] Conv2DTest.testInceptionFwd_51 [ RUN ] Conv2DTest.testInceptionFwd_6 INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:36.791904 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') I0520 05:32:36.792915 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_6): 0.04s I0520 05:32:36.828422 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_6): 0.04s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:36.829755 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') I0520 05:32:36.829983 281473723757440 conv_ops_test.py:3208] Testing InceptionFwd ([4, 8, 8, 204], [1, 1, 204, 44], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_6): 0.01s I0520 05:32:36.843556 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_6): 0.01s [ OK ] Conv2DTest.testInceptionFwd_6 [ RUN ] Conv2DTest.testInceptionFwd_No_Winograd_Nonfused INFO:tensorflow:Running Test in GRAPH mode. I0520 05:32:36.844732 281473723757440 test_util.py:1494] Running Test in GRAPH mode. INFO:tensorflow:Skipping InceptionFwd ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') I0520 05:32:36.845829 281473723757440 conv_ops_test.py:3205] Skipping InceptionFwd ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_No_Winograd_Nonfused): 0.0s I0520 05:32:36.846089 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_No_Winograd_Nonfused): 0.0s INFO:tensorflow:Running Test in EAGER mode. I0520 05:32:36.846723 281473723757440 test_util.py:1503] Running Test in EAGER mode. INFO:tensorflow:Skipping InceptionFwd ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') I0520 05:32:36.846954 281473723757440 conv_ops_test.py:3205] Skipping InceptionFwd ([1, 400, 400, 1], [1, 1, 1, 256], 1, 'SAME') INFO:tensorflow:time(__main__.Conv2DTest.testInceptionFwd_No_Winograd_Nonfused): 0.0s I0520 05:32:36.847213 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInceptionFwd_No_Winograd_Nonfused): 0.0s [ OK ] Conv2DTest.testInceptionFwd_No_Winograd_Nonfused [ RUN ] Conv2DTest.testInputGradient1x1PaddingStrideTwo INFO:tensorflow:time(__main__.Conv2DTest.testInputGradient1x1PaddingStrideTwo): 0.35s I0520 05:32:37.199123 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInputGradient1x1PaddingStrideTwo): 0.35s [ OK ] Conv2DTest.testInputGradient1x1PaddingStrideTwo [ RUN ] Conv2DTest.testInputGradientKernelSizeMatchesInputSize INFO:tensorflow:time(__main__.Conv2DTest.testInputGradientKernelSizeMatchesInputSize): 0.29s I0520 05:32:37.489304 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInputGradientKernelSizeMatchesInputSize): 0.29s [ OK ] Conv2DTest.testInputGradientKernelSizeMatchesInputSize [ RUN ] Conv2DTest.testInputGradientValidPaddingStrideOne INFO:tensorflow:time(__main__.Conv2DTest.testInputGradientValidPaddingStrideOne): 0.66s I0520 05:32:38.153149 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testInputGradientValidPaddingStrideOne): 0.66s [ OK ] Conv2DTest.testInputGradientValidPaddingStrideOne [ RUN ] Conv2DTest.testShapeFunctionEdgeCases INFO:tensorflow:time(__main__.Conv2DTest.testShapeFunctionEdgeCases): 0.02s I0520 05:32:38.178099 281473723757440 test_util.py:2464] time(__main__.Conv2DTest.testShapeFunctionEdgeCases): 0.02s [ OK ] Conv2DTest.testShapeFunctionEdgeCases [ RUN ] DeepConv2DTest.test_session [ SKIPPED ] DeepConv2DTest.test_session [ RUN ] FusedConv2DTest.testAddWithRefCountTwoAndNoDependence INFO:tensorflow:Running testAddWithRefCountTwoAndNoDependence in GRAPH mode. I0520 05:32:38.179556 281473723757440 test_util.py:1494] Running testAddWithRefCountTwoAndNoDependence in GRAPH mode. INFO:tensorflow:time(__main__.FusedConv2DTest.testAddWithRefCountTwoAndNoDependence): 0.05s I0520 05:32:38.232736 281473723757440 test_util.py:2464] time(__main__.FusedConv2DTest.testAddWithRefCountTwoAndNoDependence): 0.05s INFO:tensorflow:Running testAddWithRefCountTwoAndNoDependence in EAGER mode. I0520 05:32:38.234230 281473723757440 test_util.py:1503] Running testAddWithRefCountTwoAndNoDependence in EAGER mode. INFO:tensorflow:time(__main__.FusedConv2DTest.testAddWithRefCountTwoAndNoDependence): 0.02s I0520 05:32:38.257486 281473723757440 test_util.py:2464] time(__main__.FusedConv2DTest.testAddWithRefCountTwoAndNoDependence): 0.02s [ OK ] FusedConv2DTest.testAddWithRefCountTwoAndNoDependence [ RUN ] FusedConv2DTest.testResizeAndPadLargeResize INFO:tensorflow:Running testResizeAndPadLargeResize in GRAPH mode. I0520 05:32:38.258881 281473723757440 test_util.py:1494] Running testResizeAndPadLargeResize in GRAPH mode. 2023-05-20 05:32:38.279821: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at conv_ops_fused_image_transform.cc:671 : INVALID_ARGUMENT: Encountered overflow when multiplying 5637144576 with 1879048192, result: -7854277750134145024 2023-05-20 05:32:38.279895: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: Encountered overflow when multiplying 5637144576 with 1879048192, result: -7854277750134145024 [[{{node FusedResizeAndPadConv2D}}]] ERROR:tensorflow:Graph execution error: Detected at node 'FusedResizeAndPadConv2D' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3504, in test.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.10/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.10/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.10/unittest/runner.py", line 184, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/case.py", line 650, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/case.py", line 591, in run self._callTestMethod(testMethod) File "/usr/lib/python3.10/unittest/case.py", line 549, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3449, in testResizeAndPadLargeResize gen_nn_ops.fused_resize_and_pad_conv2d( Node: 'FusedResizeAndPadConv2D' Encountered overflow when multiplying 5637144576 with 1879048192, result: -7854277750134145024 [[{{node FusedResizeAndPadConv2D}}]] Original stack trace for 'FusedResizeAndPadConv2D': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3504, in test.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.10/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.10/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.10/unittest/runner.py", line 184, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/case.py", line 650, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/case.py", line 591, in run self._callTestMethod(testMethod) File "/usr/lib/python3.10/unittest/case.py", line 549, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2335, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1498, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3449, in testResizeAndPadLargeResize gen_nn_ops.fused_resize_and_pad_conv2d( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_nn_ops.py", line 5132, in fused_resize_and_pad_conv2d _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3315, in _create_op_internal ret = Operation.from_node_def( E0520 05:32:38.953699 281473723757440 test_util.py:2067] Graph execution error: Detected at node 'FusedResizeAndPadConv2D' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3504, in test.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.10/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.10/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.10/unittest/runner.py", line 184, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/case.py", line 650, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/case.py", line 591, in run self._callTestMethod(testMethod) File "/usr/lib/python3.10/unittest/case.py", line 549, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3449, in testResizeAndPadLargeResize gen_nn_ops.fused_resize_and_pad_conv2d( Node: 'FusedResizeAndPadConv2D' Encountered overflow when multiplying 5637144576 with 1879048192, result: -7854277750134145024 [[{{node FusedResizeAndPadConv2D}}]] Original stack trace for 'FusedResizeAndPadConv2D': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3504, in test.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.10/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.10/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.10/unittest/runner.py", line 184, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.10/unittest/case.py", line 650, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.10/unittest/case.py", line 591, in run self._callTestMethod(testMethod) File "/usr/lib/python3.10/unittest/case.py", line 549, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2335, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1498, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 3449, in testResizeAndPadLargeResize gen_nn_ops.fused_resize_and_pad_conv2d( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_nn_ops.py", line 5132, in fused_resize_and_pad_conv2d _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3315, in _create_op_internal ret = Operation.from_node_def( INFO:tensorflow:time(__main__.FusedConv2DTest.testResizeAndPadLargeResize): 0.7s I0520 05:32:38.955909 281473723757440 test_util.py:2464] time(__main__.FusedConv2DTest.testResizeAndPadLargeResize): 0.7s INFO:tensorflow:Running testResizeAndPadLargeResize in EAGER mode. I0520 05:32:38.957199 281473723757440 test_util.py:1503] Running testResizeAndPadLargeResize in EAGER mode. 2023-05-20 05:32:38.968181: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at conv_ops_fused_image_transform.cc:671 : INVALID_ARGUMENT: Encountered overflow when multiplying 5637144576 with 1879048192, result: -7854277750134145024 INFO:tensorflow:time(__main__.FusedConv2DTest.testResizeAndPadLargeResize): 0.01s I0520 05:32:38.968785 281473723757440 test_util.py:2464] time(__main__.FusedConv2DTest.testResizeAndPadLargeResize): 0.01s [ OK ] FusedConv2DTest.testResizeAndPadLargeResize [ RUN ] SeparableConv2DTest.testSeparableConv2DEqualInputOutputDepthNCHW INFO:tensorflow:time(__main__.SeparableConv2DTest.testSeparableConv2DEqualInputOutputDepthNCHW): 0.0s I0520 05:32:38.970298 281473723757440 test_util.py:2464] time(__main__.SeparableConv2DTest.testSeparableConv2DEqualInputOutputDepthNCHW): 0.0s [ OK ] SeparableConv2DTest.testSeparableConv2DEqualInputOutputDepthNCHW ====================================================================== FAIL: testConv2DGroupConvFwd (__main__.Conv2DTest) Conv2DTest.testConv2DGroupConvFwd ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2335, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1498, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 852, in testConv2DGroupConvFwd self._VerifyGroupConvFwd([10, 32, 32, 16], filter_dims, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/nn_ops/conv_ops_test.py", line 839, in _VerifyGroupConvFwd self.assertAllClose( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1539, in decorated return f(*args, **kwds) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3172, in assertAllClose self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3128, in _assertAllCloseRecursive self._assertArrayLikeAllClose( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3065, in _assertArrayLikeAllClose np.testing.assert_allclose( File "/usr/local/lib/python3.10/dist-packages/numpy/testing/_private/utils.py", line 1527, in assert_allclose assert_array_compare(compare, actual, desired, err_msg=str(err_msg), File "/usr/local/lib/python3.10/dist-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=1e-05, atol=1e-05 Mismatched value: a is different from b. not close where = (array([0, 0, 0, ..., 9, 9, 9]), array([0, 0, 0, ..., 7, 7, 7]), array([ 0, 0, 0, ..., 31, 31, 31]), array([ 0, 1, 2, ..., 29, 30, 31])) not close lhs = [1.0729360e+06 1.5484760e+06 1.6210520e+06 ... 3.9282451e+08 3.9286291e+08 2.2004294e+08] not close rhs = [7.2055200e+05 1.0859000e+06 1.1446520e+06 ... 5.3390678e+08 5.3395901e+08 3.4552045e+08] not close dif = [3.5238400e+05 4.6257600e+05 4.7640000e+05 ... 1.4108227e+08 1.4109610e+08 1.2547750e+08] not close tol = [ 7.2055297 10.859009 11.446529 ... 5339.068 5339.59 3455.2043 ] dtype = float32, shape = (10, 8, 32, 32) Mismatched elements: 81920 / 81920 (100%) Max absolute difference: 1.410961e+08 Max relative difference: 0.4890473 x: array([[[[1.072936e+06, 1.548476e+06, 1.621052e+06, ..., 3.580604e+06, 3.653180e+06, 2.265128e+06], [2.758524e+06, 3.885114e+06, 3.966330e+06, ..., 6.159162e+06,... y: array([[[[7.205520e+05, 1.085900e+06, 1.144652e+06, ..., 2.730956e+06, 2.789708e+06, 1.830760e+06], [2.017836e+06, 2.994042e+06, 3.075258e+06, ..., 5.268090e+06,... ---------------------------------------------------------------------- Ran 84 tests in 17.251s FAILED (failures=1, skipped=2) ================================================================================ //tensorflow/c:c_api_experimental_test PASSED in 41.1s //tensorflow/c:c_api_function_test PASSED in 47.2s //tensorflow/c:c_api_test_cpu PASSED in 45.5s //tensorflow/c:c_test PASSED in 40.3s //tensorflow/c:env_test_cpu PASSED in 34.6s //tensorflow/c:kernels_test_cpu PASSED in 49.5s //tensorflow/c:ops_test PASSED in 35.0s //tensorflow/c:while_loop_test PASSED in 35.8s //tensorflow/c/eager:c_api_cluster_test_cpu PASSED in 40.2s //tensorflow/c/eager:c_api_remote_function_test_cpu PASSED in 40.1s //tensorflow/c/eager:c_api_remote_test_cpu PASSED in 37.1s //tensorflow/c/eager:c_api_test_cpu PASSED in 41.6s //tensorflow/c/eager:custom_device_test PASSED in 44.8s //tensorflow/c/eager/parallel_device:parallel_device_lib_test PASSED in 36.7s //tensorflow/c/eager/parallel_device:parallel_device_remote_test PASSED in 37.3s //tensorflow/c/eager/parallel_device:parallel_device_test PASSED in 42.0s //tensorflow/c/experimental/filesystem/plugins/gcs:expiring_lru_cache_test PASSED in 0.4s //tensorflow/c/experimental/filesystem/plugins/gcs:ram_file_block_cache_test PASSED in 2.9s //tensorflow/c/experimental/grappler:grappler_test PASSED in 35.6s //tensorflow/c/experimental/ops/gen/common:case_format_test PASSED in 1.2s //tensorflow/c/experimental/ops/gen/cpp:cpp_generator_test PASSED in 0.7s //tensorflow/c/experimental/ops/gen/cpp/renderers:renderer_test PASSED in 0.9s //tensorflow/c/experimental/saved_model/core:constant_loading_test PASSED in 29.9s //tensorflow/c/experimental/saved_model/core:object_graph_traversal_test PASSED in 15.5s //tensorflow/c/experimental/saved_model/core:saved_variable_loading_test PASSED in 27.8s //tensorflow/c/experimental/saved_model/core:signature_flattening_test PASSED in 22.0s //tensorflow/c/experimental/saved_model/core:tf_concrete_function_loading_test PASSED in 18.4s //tensorflow/c/experimental/saved_model/core/ops:restore_ops_test PASSED in 24.0s //tensorflow/c/experimental/saved_model/core/ops:variable_ops_test PASSED in 16.7s //tensorflow/c/experimental/saved_model/internal:saved_model_api_test PASSED in 48.8s //tensorflow/c/experimental/stream_executor:stream_executor_test PASSED in 0.3s //tensorflow/c/kernels:bitcast_op_test PASSED in 1.2s //tensorflow/c/kernels:summary_op_benchmark_test PASSED in 1.0s //tensorflow/c/kernels:summary_op_test PASSED in 1.2s //tensorflow/c/kernels:tensor_shape_utils_test PASSED in 0.2s //tensorflow/cc:cc_op_gen_test PASSED in 0.3s //tensorflow/cc:client_client_session_test PASSED in 3.8s //tensorflow/cc:coordinator_test PASSED in 6.1s //tensorflow/cc:framework_cc_ops_test PASSED in 4.7s //tensorflow/cc:framework_gradient_checker_test PASSED in 3.7s //tensorflow/cc:framework_gradients_test PASSED in 7.5s //tensorflow/cc:framework_scope_test PASSED in 1.2s //tensorflow/cc:framework_while_gradients_test PASSED in 4.2s //tensorflow/cc:gradients_array_grad_test PASSED in 20.5s //tensorflow/cc:gradients_data_flow_grad_test PASSED in 4.6s //tensorflow/cc:gradients_functional_grad_test PASSED in 2.6s //tensorflow/cc:gradients_image_grad_test PASSED in 9.6s //tensorflow/cc:gradients_linalg_grad_test PASSED in 5.1s //tensorflow/cc:gradients_manip_grad_test PASSED in 2.4s //tensorflow/cc:gradients_math_grad_test PASSED in 13.3s //tensorflow/cc:gradients_nn_grad_test PASSED in 25.7s //tensorflow/cc:gradients_resource_variable_grad_test PASSED in 3.1s //tensorflow/cc:ops_const_op_test PASSED in 0.6s //tensorflow/cc:ops_while_loop_test PASSED in 2.7s //tensorflow/cc:queue_runner_test PASSED in 13.0s //tensorflow/cc/experimental/base/tests:tensor_test PASSED in 0.2s //tensorflow/cc/experimental/base/tests:tensorhandle_test PASSED in 39.7s //tensorflow/cc/experimental/libexport:load_test PASSED in 0.3s //tensorflow/cc/experimental/libexport:save_test PASSED in 0.4s //tensorflow/cc/experimental/libtf:libtf_module_test PASSED in 38.4s //tensorflow/cc/experimental/libtf:libtf_object_test PASSED in 0.2s //tensorflow/cc/experimental/libtf:libtf_perf_test PASSED in 0.9s //tensorflow/cc/experimental/libtf:libtf_runtime_test PASSED in 41.5s //tensorflow/cc/experimental/libtf:libtf_transform_test PASSED in 38.5s //tensorflow/cc/experimental/libtf:libtf_value_test PASSED in 0.2s //tensorflow/cc/experimental/libtf:libtf_visit_test PASSED in 0.2s //tensorflow/cc/experimental/libtf/impl:iostream_test PASSED in 2.0s //tensorflow/cc/experimental/libtf/impl:none_test PASSED in 0.5s //tensorflow/cc/experimental/libtf/impl:scalars_test PASSED in 0.7s //tensorflow/cc/experimental/libtf/impl:string_test PASSED in 0.8s //tensorflow/cc/experimental/libtf/impl:tensor_spec_test PASSED in 0.2s //tensorflow/cc/saved_model:bundle_v2_test PASSED in 0.4s //tensorflow/cc/saved_model:fingerprinting_test PASSED in 2.3s //tensorflow/cc/saved_model:metrics_test PASSED in 0.2s //tensorflow/cc/saved_model:reader_test PASSED in 0.1s //tensorflow/cc/saved_model:saved_model_bundle_lite_test PASSED in 7.8s //tensorflow/cc/saved_model:saved_model_bundle_test PASSED in 12.2s //tensorflow/cc/saved_model:util_test PASSED in 0.7s //tensorflow/cc/saved_model/experimental/tests:saved_model_api_test PASSED in 45.4s //tensorflow/cc/tools:freeze_saved_model_test PASSED in 2.4s //tensorflow/compiler/aot:codegen_test PASSED in 43.0s //tensorflow/compiler/jit:compilability_check_util_test PASSED in 20.6s //tensorflow/compiler/jit:deadness_analysis_test PASSED in 12.7s //tensorflow/compiler/jit:device_compilation_cache_test PASSED in 6.2s //tensorflow/compiler/jit:device_compilation_cluster_signature_test PASSED in 19.0s //tensorflow/compiler/jit:device_compilation_profiler_test PASSED in 25.5s //tensorflow/compiler/jit:device_compiler_client_test PASSED in 5.1s //tensorflow/compiler/jit:device_compiler_disable_test PASSED in 24.0s //tensorflow/compiler/jit:device_executable_persistor_test PASSED in 30.3s //tensorflow/compiler/jit:device_util_test PASSED in 6.7s //tensorflow/compiler/jit:encapsulate_util_test PASSED in 1.1s //tensorflow/compiler/jit:node_matchers_test PASSED in 0.9s //tensorflow/compiler/jit:resource_operation_safety_analysis_test PASSED in 14.6s //tensorflow/compiler/jit:shape_inference_test PASSED in 1.1s //tensorflow/compiler/jit:xla_activity_listener_test PASSED in 35.0s //tensorflow/compiler/jit:xla_cluster_util_test PASSED in 13.4s //tensorflow/compiler/jit:xla_compile_util_test PASSED in 7.5s //tensorflow/compiler/jit:xla_kernel_creator_test PASSED in 9.6s //tensorflow/compiler/jit:xla_launch_util_test PASSED in 27.5s //tensorflow/compiler/jit/tests:auto_clustering_test PASSED in 31.9s //tensorflow/compiler/mlir:mlir_graph_optimization_pass_test PASSED in 13.8s //tensorflow/compiler/mlir:register_common_dialects_test PASSED in 20.6s //tensorflow/compiler/mlir/lite:lstm_utils_test PASSED in 2.1s //tensorflow/compiler/mlir/lite:perception_ops_utils_test PASSED in 1.0s //tensorflow/compiler/mlir/lite:size_utils_test PASSED in 0.3s //tensorflow/compiler/mlir/lite:tftext_utils_test PASSED in 0.6s //tensorflow/compiler/mlir/lite/experimental/remat:rematerializer_test PASSED in 1.5s //tensorflow/compiler/mlir/lite/experimental/tac:execution_metadata_exporter_test PASSED in 5.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests:compute-cost.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-gpu.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-nnapi.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/experimental/tac/tests:fold-constants-to-subgraph.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-alternative-subgraph.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-op-cost.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/experimental/tac/tests:pick-subgraphs.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/experimental/tac/tests:raise-target-subgraphs.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/experimental/tac/tests:tac-filter.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/experimental/tac/tests:target-annotation.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:device-transform-nnapi.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:simple-graph.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/metrics:error_collector_inst_test PASSED in 0.9s //tensorflow/compiler/mlir/lite/quantization:numerical_utils_test PASSED in 0.1s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_model_test PASSED in 17.8s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_weights_test PASSED in 12.5s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_default.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_legacy.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant_4bit.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/quantization/tests:import_quant_stats.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/sparsity:sparsify_model_test PASSED in 1.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:fold_broadcast.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:fuse_mhlo_convolution.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-inplaceupdate.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-skip-quantization-ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tf-fb-tf.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-add.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-broadcast_in_dim.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-clamp.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-compare.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-concat.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-constant.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-conv.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-dot.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-gather.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-max.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-mul.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-pad.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-reshape.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-rsqrt.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-scatter.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-sub.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-add.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-broadcast.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-clamp.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-concat.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-constant.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-conv.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-max.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-mul.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-pad.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-reshape.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-rsqrt.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-sub.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-allow-tf.mlir.test PASSED in 2.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-smuggle-resize.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:optimize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-clamp.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-concat.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-conv.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-division.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-logistic.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-multiply.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-reduce-window.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-resize-bilinear.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-subtract.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-tf-quantize.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:unfuse_mhlo_batch_norm.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:analyze-variables.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:canonicalize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:decompose-hybrid-quantization.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:default_quant_params.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:dilated-conv.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:fuse-tftext.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:get-arithmetic-count.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests:guarantee_func_has_one_use.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:inlining.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:insert_call_once_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:legalize-tf-assert.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:legalize-tf-hashtables.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:legalize-tf-no-runtime-verification.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:legalize-tf-variables.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests:legalize-tf-while.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:legalize-tf.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/tests:legalize_jax_random.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/tests:lift_tflite_flex_ops.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-default-to-single-batch.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-enable-dynamic-update-slice.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests:modify_io_nodes.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests:ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:optimize-after-quantization.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:optimize.mlir.test PASSED in 4.4s //tensorflow/compiler/mlir/lite/tests:optimize_functional_ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:optimize_no_verify.mlir.test PASSED in 3.1s //tensorflow/compiler/mlir/lite/tests:optimize_op_order.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:partitioned-topological-sort.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:pin-ops-with-side-effects.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:post-quantize-dynamic-range.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:post-quantize.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests:prepare-composite-functions-tf.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-dynamic-range.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training-16bits.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-signed.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:prepare-quantize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant-4bit.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:prepare-tf-with-allowing-bf16-and-f16-type-legalization.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:prepare-tf.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:quantize-dynamic-range.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests:quantize-numeric-verify.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/lite/tests:quantize-variables.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:quantize.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:raise-custom-ops.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:reduce_while_operands.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:shape-inference.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:split-merged-operands.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:tfl_while_op_licm.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:tfl_while_outline.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests:trim-functions-tf.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:unfold-large-splat-constant.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.line.part.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.stack.part.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests/end2end:add.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:back2back_fake_quant.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:control_flow_v1.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d_nchw.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:custom_opdef.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/end2end:disallow_stateful_partitioned_call.pbtxt.test PASSED in 2.2s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel_4bit.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity.pbtxt.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity_4bit.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:graph-input-node.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/end2end:graph_with_placeholder_with_default.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:if_op.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:quant_stats.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul_disabled.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:basic_lstm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:bucketize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:control_edges.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:dynamic_shape.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:empty_input_output_names.json.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:external_constant.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:if_op.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:import_json.json.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_arrays.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_output_names_attr.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:legacy_reshape.json.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.json.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:many_attribute_op.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:math.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:matmul.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:multi_output_op.json.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional_input.json.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:output_arrays.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning_function_input_as_output.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quant_stats.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quantization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:reshape.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature_with_multiple_entry_points.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:simple.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:tf_variant_type.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_function_output.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_tensor.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:while_op.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests/mlir2exec:tfl_while_op.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:basic_lstm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:bucketize.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_op_with_tflite_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d_v2.mlir.test PASSED in 3.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_builtin.mlir.test PASSED in 3.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_custom.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex_enable_builtin.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:dynamic_shape_constant.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fake_quant.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_exclusively.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_complex128.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_f64.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_tflite_op.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected_v2.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:hashtable_resource.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:if_op.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:logical.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:low_bit_packing.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_asym_attr.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_quantized.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:math.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:metadata.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v2.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v3.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:nn.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:numeric_verify.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:optional.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:quantization.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:reshape.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_output_override.mlir.test PASSED in 0.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_multiple_entry_points.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_no_inputs.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_connected_control_nodes.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_unconnected_control_nodes.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf_v2.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tf_entry_function.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tfl_while_op.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:transpose_conv_optional.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:type_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_lstm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_rnn.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unranked_tensor.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unsorted_segment_prod.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_func.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:while_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/quantization/stablehlo/tests:fill_quantization_options_test PASSED in 0.4s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibrator_singleton_test PASSED in 0.5s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:custom_aggregator_op_test PASSED in 25.2s //tensorflow/compiler/mlir/quantization/tensorflow/cc:const_op_size_test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/cc:convert_asset_args_test PASSED in 7.0s //tensorflow/compiler/mlir/quantization/tensorflow/cc:save_variables_test PASSED in 0.5s //tensorflow/compiler/mlir/quantization/tensorflow/cc:status_macro_test PASSED in 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/debugging:mlir_dump_test PASSED in 0.5s //tensorflow/compiler/mlir/quantization/tensorflow/python:concurrency_test PASSED in 39.3s //tensorflow/compiler/mlir/quantization/tensorflow/python:pywrap_quantize_model_test PASSED in 12.5s //tensorflow/compiler/mlir/quantization/tensorflow/python:representative_dataset_test PASSED in 8.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:cast_bf16_ops_to_f32.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_custom_aggregation_op_to_quant_stats.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_fake_quant_to_qdq.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tf_quant_ops_to_mhlo.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tpu_model_to_cpu.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:duplicate_shape_determining_constants.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_flow.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_xla.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_custom_aggregation_ops.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_main_function.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_drq.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_weight_only.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_restore_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_save_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:issue_ids_of_custom_aggregation_ops.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq_min_elements.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_xla.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:mark_functions_noinline.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_initializer_function_ops_to_main.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_save_function_ops_to_main.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:optimize.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_lifting.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq_per_channel.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq_per_channel.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op_weight_only.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_drq.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_weight_only.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_xla.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_drq.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_xla.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:remove_var_init_by_const.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops.mlir.test PASSED in 2.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops_large_constants.mlir.test PASSED in 22.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:unfreeze_constants.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/utils:tf_to_xla_attribute_utils_test PASSED in 45.0s //tensorflow/compiler/mlir/stablehlo:stablehlo_test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow:bridge_logger_test PASSED in 11.6s //tensorflow/compiler/mlir/tensorflow:cluster_util_test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow:convert_tensor_test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow:convert_type_test PASSED in 0.2s //tensorflow/compiler/mlir/tensorflow:device_util_test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow:dump_graph_test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow:dump_mlir_util_test PASSED in 13.2s //tensorflow/compiler/mlir/tensorflow:error_util_test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow:tf_saved_model_test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow:tpu_rewrite_device_util_test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:add_functions_for_exported_names.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:annotate-parameter-replication.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:batchmatmul_to_einsum.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:breakup-islands.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:cannonicalize_ops_outside_compilation.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize_compile_and_replicate_attributes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:check_control_dependencies.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:cluster_formation.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:cluster_ops_by_policy.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/tensorflow/tests:cluster_outlining.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:cluster_tf_ops_pass.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests:constant-fold.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:constant_op_device_assignment.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:convert-tf-control-flow-to-scf.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:convert_control_to_data_outputs.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:convert_launch_func_to_tf_call.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:convert_session_initializer_to_function.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:convert_to_legacy_compile_and_replicate_attributes.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:decompose_reduce_dataset.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:decompose_resource_ops.mlir.test PASSED in 2.6s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment_by_func_attr.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:device_attribute_to_launch.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests:device_canonicalize.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:device_copy.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:drop_while_shape_invariant.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:einsum.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:embedding_pipelining.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:empty-main.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:end-to-end-tpu-reshard-variables.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:executor_canonicalize.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_coarsening.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_materialize_const.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:extract_head_tail_outside_compilation.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:extract_outside_compilation.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:extract_tpu_copy_with_dynamic_shape_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:fold-broadcast.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:freeze_variables.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:func-attr-invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:func-attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-cfg.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-regions.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if-fail.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:fused_kernel_matcher.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:gpu_fusion.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning_preserve_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:group_by_dialect.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:guarantee-all-funcs-one-use.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:hoist_loop_invariant.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:hoist_replicate_invariant_resource_writes.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:host_launch_to_outside_compiled.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_invalid.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_saved_model.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:inlining.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:isolate-placer.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:launch_outlining.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute_legacy.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_60.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_70.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nchw.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nhwc.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_begin.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_end.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nchw.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nhwc.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:legalize_hlo.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_arg_control_dep.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_with_control_flow.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:localize_var_handles.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program_invalid.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:lower_quantized.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:lower_tf.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:lower_variable_ops_to_ml_program.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:mark_input_output_aliases.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:mark_ops_for_outside_compilation.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:materialize_passthrough_op.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:merge_control_flow.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests:mlprogram.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:name_anonymous_iterators.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:optimize-arg-operand-constraint.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:optimize.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:order_by_dialect.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:outside_compiled_to_host_launch.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands_legacy.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:prepare_tpu_computation_for_tf_export.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args_functions.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:promote_var_handles_to_args.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:readonly_references_to_resources.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:region-control-flow-to-functional.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_arguments.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_while_results.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:replica_id_to_device_ordinal.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:replicate_invariant_op_hoisting.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:replicate_tensor_list_init_ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island_legacy.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:resource-alias-analysis-test.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:resource-device-inference.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:resource_analyzer.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:resource_inlining.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:resource_op_lifting.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:rewrite_tpu_embedding_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:roundtrip-tf-executor.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:shape_inference.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:side-effect-analysis-test.mlir.test PASSED in 2.7s //tensorflow/compiler/mlir/tensorflow/tests:sink_constant.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:split_into_island_per_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:stack_ops_decomposition.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:strip_noinline.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:strip_saved_module_metadata.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:strip_tf_attributes.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tensor_array_ops_decomposition.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tensor_list_ops_decomposition.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:tf-executor-to-functional.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf-functional-to-executor.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf-ops.mlir.test PASSED in 6.0s //tensorflow/compiler/mlir/tensorflow/tests:tf-reduce-identity.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_map_and_batch.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_pmap_and_batch.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_index_selector.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops_invalid.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_location_roundtrip.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_printer.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_side_effect.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_optimize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_deduplicate_bound_input_bindings.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_assets.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors_mutable_tensors.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init_fail.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables_invalid_session.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_mark_initialized_variables.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops_invalid.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors_interprocedural.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_remove_vars_in_session_initializer.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tf_side_effect.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_trait_folds.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu-annotate-dynamic-shape-inputs.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu-cluster-cleanup-attributes.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu-dynamic-layout-pass.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu-merge-variables-with-execute.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tpu-multiple-while-body-func.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu-resource-read-for-write.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tpu-variable-runtime-reformatting.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu_cluster_formation.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_composite_resource_ops.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_splits.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tpu_device_propagation.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tpu_host_computation_expansion.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tpu_identity_pruning.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_parallel_execute_sink_resource_write.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_partitioned_op_conversion.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu_reorder_replicate_and_partitioned_inputs.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_resource_partitioning.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tpu_rewrite.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tpu_sharding_identification.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_space_to_depth_pass.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tpu_tail_with_tobool_op.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_update_embedding_enqueue_op_inputs.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_validate_inputs.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:transpose-op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:unroll-batch-matmul.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:update_control_dependencies.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:warn_when_using_deprecated_dumps.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:while_licm.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_deserialization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_round_trip.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_serialization.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:xla_cluster_formation.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests:xla_inline_device_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:add.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding-invalid.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding-hook.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:mlir-module-serialized-str-attr.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:replicate-tensor-list-init-ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:result-sharding.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr-invalid.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference-after-legalization.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:stablehlo_add.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:executor_tpuv1_island_coarsening.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:while_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:executor_tpuv1_inline_tpu_island.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:while_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:case_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:executor_tpuv1_outline_tpu_island.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:while_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:add.pbtxt.test PASSED in 2.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-as-fetch.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-control-dep.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type-with-subtype.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type.pbtxt.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-multi-data-type-with-subtype.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-retval-attrs.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:case_op.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:const-values.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:device-arg-retval-attr.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-input-shapes.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-value-attr.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-as-fetch.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-control-dep.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:force_shared_name_for_resource_ops.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:function-func-attr.pbtxt.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-if-ops.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-while-ops.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-control-ret.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-retval-of-arg.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-custom-operation.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-default-attr.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-device-retval.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-empty-tensor-content.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-func-attr.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-call.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-diff-island.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-same-island.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-defs.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-input-shapes.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-name-bug.pbtxt.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-resource-args.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-gradient-def.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-input-func-arg-name-collision.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-library.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-malformed.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-scalar-input.pbtxt.test PASSED in 2.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-uint8-return.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-undefined-output.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-version-info.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-while-loop.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:invalid-output-index.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:legacy-fed-input-without-inputs.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:merge_node_with_function.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:mlir_passthrough_op.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multi-output-feeds.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multiple-use-next-iteration.pbtxt.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:node-locations.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes-attr.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example_v2.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:partial-device-name.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:prune_unused_nodes.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:quint8-const.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:shape-attrs.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:stateful-attribute.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:string-attr.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:switch_n.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:target.pbtxt.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tensor-list.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tf-data-pipeline.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:unregistered_kernel.pbtxt.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/batch_use_same_function:saved_model.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:aliasing_arg_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:case.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:convert_tensor.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_shape_attr.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_size_attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:device-arg-retval-attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:export_main_to_flib.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:fetch_feed_names.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_list_attr.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-control-ret.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-order.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args-handle-info.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-if-ops.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-while-ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:graph-as-function.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:infer_derived_attribute.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:invalid_input.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:legalized_name.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:missing-main.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:noop.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:optional_symbol_ref.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:output-shapes-attr.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example_v2.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:preserve-entry-func-names.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-type-attr.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-while-loop.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:shape_list_attr.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple_tf_dialect_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:stringescape.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:switchn.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-gradient-attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-legacy-call.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_add.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_identity_n.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_tpu_embedding_ops.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_attr.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_list_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_name.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_output_name.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:while-loop.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/tf_to_hlo_pipeline:sccp-post-shape-inference.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/tpu_bridge_v1:end_to_end.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tf2xla/api/v0:compile_mlir_util_test PASSED in 8.9s //tensorflow/compiler/mlir/tf2xla/api/v0:compile_tf_graph_test PASSED in 0.2s //tensorflow/compiler/mlir/tf2xla/api/v1:legalize_tf_test PASSED in 26.0s //tensorflow/compiler/mlir/tf2xla/tests:adjust-layout.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:convert-mhlo-quant-to-int.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_runtime_pipeline.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_sparsification.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-BatchMatMulV2.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-binary-elementwise.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-collective.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-communication.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-include-tf2xla-fallback.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-no-tf2xla-fallback.mlir.test PASSED in 5.1s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-prefer-tf2xla.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-types.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-with-tf2xla-hlo-importer.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-with-tf2xla.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf.mlir.test PASSED in 12.6s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_cpu.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_gpu.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization-no-chlo.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tf2xla/transforms:tf2xla_rewriter_test PASSED in 15.9s //tensorflow/compiler/mlir/tf2xla/transforms:verify_tfxla_legalization_test PASSED in 17.5s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_targets_test PASSED in 1.5s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_tf_test PASSED in 2.7s //tensorflow/compiler/mlir/tfr:graph_decompose_test PASSED in 13.9s //tensorflow/compiler/mlir/tfr:node_expansion_test PASSED in 7.4s //tensorflow/compiler/mlir/tfr:op_reg_gen_test PASSED in 17.3s //tensorflow/compiler/mlir/tfr:tfr_decompose_ctx_test PASSED in 9.1s //tensorflow/compiler/mlir/tfr:tfr_gen_test PASSED in 14.9s //tensorflow/compiler/mlir/tfr/examples/customization:test_ops_test PASSED in 20.7s //tensorflow/compiler/mlir/tfr/examples/pad:pad_ops_test PASSED in 30.3s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_deallocation.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_reuse.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tools/kernel_gen/tests:bufferize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:copy_cleanup.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:embed_tf_framework.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:invalid.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tools/kernel_gen/tests:isinf.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:ops.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tools/kernel_gen/tests:parallel_loops_to_sequential.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tools/kernel_gen/tests:rewrite_tf_framework_assert.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tanh.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf-legalize-to-lmhlo.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_abi_knowledge.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_framework_legalize_to_llvm.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_kernel_gpu_launch_to_llvm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_to_jit_invocations.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tosa/tests:convert-tfl-uint8.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:convert_metadata.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:fuse-bias-tf.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:lower-complex-types.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:lower_global_tensors.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tosa/tests:multi_add.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:retain_call_once_funcs.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:strip-quant-types.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tosa/tests:strip_metadata.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:tf-tfl-to-tosa-pipeline.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tosa/tests:tf-to-tosa-pipeline.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-dequantize_softmax.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline-filtered.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline.mlir.test PASSED in 10.5s //tensorflow/compiler/mlir/tosa/tests:verify_fully_converted.mlir.test PASSED in 1.5s //tensorflow/compiler/tests:adadelta_test_cpu PASSED in 16.1s //tensorflow/compiler/tests:adagrad_da_test_cpu PASSED in 10.5s //tensorflow/compiler/tests:adagrad_test_cpu PASSED in 10.9s //tensorflow/compiler/tests:adam_test_cpu PASSED in 13.1s //tensorflow/compiler/tests:add_n_test_cpu PASSED in 7.7s //tensorflow/compiler/tests:argminmax_test_cpu PASSED in 22.2s //tensorflow/compiler/tests:argminmax_test_cpu_mlir_bridge_test PASSED in 21.5s //tensorflow/compiler/tests:bucketize_op_test_cpu PASSED in 11.9s //tensorflow/compiler/tests:bucketize_op_test_cpu_mlir_bridge_test PASSED in 9.3s //tensorflow/compiler/tests:case_test_cpu PASSED in 9.2s //tensorflow/compiler/tests:cast_ops_test_cpu PASSED in 7.6s //tensorflow/compiler/tests:cast_ops_test_cpu_mlir_bridge_test PASSED in 8.5s //tensorflow/compiler/tests:categorical_op_test_cpu PASSED in 12.1s //tensorflow/compiler/tests:categorical_op_test_cpu_mlir_bridge_test PASSED in 18.8s //tensorflow/compiler/tests:cholesky_op_test_cpu PASSED in 23.2s //tensorflow/compiler/tests:cholesky_op_test_cpu_mlir_bridge_test PASSED in 20.7s //tensorflow/compiler/tests:clustering_test_cpu PASSED in 7.8s //tensorflow/compiler/tests:clustering_test_cpu_mlir_bridge_test PASSED in 9.1s //tensorflow/compiler/tests:concat_ops_test_cpu PASSED in 10.2s //tensorflow/compiler/tests:concat_ops_test_cpu_mlir_bridge_test PASSED in 12.3s //tensorflow/compiler/tests:cond_test_cpu PASSED in 10.6s //tensorflow/compiler/tests:const_arg_test_cpu PASSED in 10.9s //tensorflow/compiler/tests:const_test_cpu PASSED in 8.6s //tensorflow/compiler/tests:data_format_ops_test_cpu PASSED in 11.3s //tensorflow/compiler/tests:data_format_ops_test_cpu_mlir_bridge_test PASSED in 14.6s //tensorflow/compiler/tests:dense_layer_test_cpu PASSED in 12.4s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu PASSED in 10.3s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu_mlir_bridge_test PASSED in 12.3s //tensorflow/compiler/tests:dynamic_stitch_test_cpu PASSED in 8.1s //tensorflow/compiler/tests:dynamic_stitch_test_cpu_mlir_bridge_test PASSED in 7.0s //tensorflow/compiler/tests:eager_test_cpu PASSED in 17.1s //tensorflow/compiler/tests:einsum_op_test_cpu PASSED in 8.2s //tensorflow/compiler/tests:einsum_op_test_cpu_mlir_bridge_test PASSED in 8.4s //tensorflow/compiler/tests:ensure_shape_op_test_cpu PASSED in 7.6s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu PASSED in 8.7s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu_mlir_bridge_test PASSED in 8.9s //tensorflow/compiler/tests:fake_quant_ops_test_cpu PASSED in 15.5s //tensorflow/compiler/tests:fake_quant_ops_test_cpu_mlir_bridge_test PASSED in 18.3s //tensorflow/compiler/tests:fifo_queue_test_cpu PASSED in 8.8s //tensorflow/compiler/tests:fifo_queue_test_cpu_mlir_bridge_test PASSED in 9.0s //tensorflow/compiler/tests:ftrl_ops_test_cpu PASSED in 9.7s //tensorflow/compiler/tests:ftrl_ops_test_cpu_mlir_bridge_test PASSED in 10.3s //tensorflow/compiler/tests:function_test_cpu PASSED in 8.3s //tensorflow/compiler/tests:function_test_cpu_mlir_bridge_test PASSED in 7.7s //tensorflow/compiler/tests:gather_nd_op_test_cpu PASSED in 11.4s //tensorflow/compiler/tests:gather_nd_op_test_cpu_mlir_bridge_test PASSED in 8.6s //tensorflow/compiler/tests:gather_test_cpu PASSED in 43.6s //tensorflow/compiler/tests:gather_test_cpu_mlir_bridge_test PASSED in 59.6s //tensorflow/compiler/tests:jit_test_cpu PASSED in 52.6s //tensorflow/compiler/tests:listdiff_op_test_cpu PASSED in 19.7s //tensorflow/compiler/tests:listdiff_op_test_cpu_mlir_bridge_test PASSED in 16.0s //tensorflow/compiler/tests:lrn_ops_test_cpu PASSED in 7.1s //tensorflow/compiler/tests:lrn_ops_test_cpu_mlir_bridge_test PASSED in 7.6s //tensorflow/compiler/tests:lstm_test_cpu PASSED in 26.7s //tensorflow/compiler/tests:manip_ops_test_cpu PASSED in 12.8s //tensorflow/compiler/tests:manip_ops_test_cpu_mlir_bridge_test PASSED in 13.3s //tensorflow/compiler/tests:matrix_band_part_test_cpu PASSED in 46.4s //tensorflow/compiler/tests:matrix_band_part_test_cpu_mlir_bridge_test PASSED in 41.9s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu PASSED in 27.3s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu_mlir_bridge_test PASSED in 24.1s //tensorflow/compiler/tests:matrix_solve_op_test_cpu PASSED in 8.3s //tensorflow/compiler/tests:matrix_solve_op_test_cpu_mlir_bridge_test PASSED in 8.2s //tensorflow/compiler/tests:matrix_triangular_solve_op_test_cpu PASSED in 37.0s //tensorflow/compiler/tests:matrix_triangular_solve_op_test_cpu_mlir_bridge_test PASSED in 33.2s //tensorflow/compiler/tests:momentum_test_cpu PASSED in 10.2s //tensorflow/compiler/tests:nary_ops_test_cpu PASSED in 9.4s //tensorflow/compiler/tests:nary_ops_test_cpu_mlir_bridge_test PASSED in 9.9s //tensorflow/compiler/tests:nullary_ops_test_cpu PASSED in 10.5s //tensorflow/compiler/tests:nullary_ops_test_cpu_mlir_bridge_test PASSED in 8.5s //tensorflow/compiler/tests:placeholder_test_cpu PASSED in 9.6s //tensorflow/compiler/tests:placeholder_test_cpu_mlir_bridge_test PASSED in 7.4s //tensorflow/compiler/tests:proximal_adagrad_test_cpu PASSED in 8.5s //tensorflow/compiler/tests:proximal_gradient_descent_test_cpu PASSED in 13.1s //tensorflow/compiler/tests:quantized_ops_test_cpu PASSED in 8.4s //tensorflow/compiler/tests:reduce_window_test_cpu PASSED in 7.4s //tensorflow/compiler/tests:reduce_window_test_cpu_mlir_bridge_test PASSED in 8.6s //tensorflow/compiler/tests:reshape_op_test_cpu PASSED in 7.6s //tensorflow/compiler/tests:reshape_op_test_cpu_mlir_bridge_test PASSED in 8.5s //tensorflow/compiler/tests:reverse_ops_test_cpu PASSED in 10.0s //tensorflow/compiler/tests:reverse_ops_test_cpu_mlir_bridge_test PASSED in 11.8s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu PASSED in 9.6s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu_mlir_bridge_test PASSED in 9.0s //tensorflow/compiler/tests:risc_ops_test_cpu_mlir_bridge_test PASSED in 9.3s //tensorflow/compiler/tests:rmsprop_test_cpu PASSED in 11.4s //tensorflow/compiler/tests:scatter_nd_op_test_cpu PASSED in 23.7s //tensorflow/compiler/tests:scatter_nd_op_test_cpu_mlir_bridge_test PASSED in 33.6s //tensorflow/compiler/tests:searchsorted_op_test_cpu PASSED in 8.6s //tensorflow/compiler/tests:searchsorted_op_test_cpu_mlir_bridge_test PASSED in 11.3s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu PASSED in 31.6s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu_mlir_bridge_test PASSED in 48.4s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu PASSED in 15.9s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu_mlir_bridge_test PASSED in 19.9s //tensorflow/compiler/tests:slice_ops_test_cpu PASSED in 16.6s //tensorflow/compiler/tests:slice_ops_test_cpu_mlir_bridge_test PASSED in 21.8s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu PASSED in 11.4s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu_mlir_bridge_test PASSED in 8.4s //tensorflow/compiler/tests:stack_ops_test_cpu PASSED in 6.9s //tensorflow/compiler/tests:tensor_float_32_test_cpu PASSED in 9.1s //tensorflow/compiler/tests:tensor_float_32_test_cpu_mlir_bridge_test PASSED in 10.6s //tensorflow/compiler/tests:tensor_list_ops_test_cpu PASSED in 11.6s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu PASSED in 13.9s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu_mlir_bridge_test PASSED in 15.4s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu PASSED in 14.7s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu_mlir_bridge_test PASSED in 14.9s //tensorflow/compiler/tests:unique_ops_test_cpu PASSED in 7.6s //tensorflow/compiler/tests:variable_ops_test_cpu PASSED in 28.5s //tensorflow/compiler/tests:variable_ops_test_cpu_mlir_bridge_test PASSED in 18.0s //tensorflow/compiler/tests:where_op_test_cpu PASSED in 9.2s //tensorflow/compiler/tests:while_test_cpu PASSED in 12.4s //tensorflow/compiler/tests:xla_call_module_test_cpu PASSED in 14.4s //tensorflow/compiler/tests:xla_custom_call_ops_test_cpu PASSED in 8.4s //tensorflow/compiler/tests:xla_device_gpu_test_cpu PASSED in 7.0s //tensorflow/compiler/tests:xla_device_test_cpu PASSED in 16.4s //tensorflow/compiler/tests:xla_device_test_cpu_mlir_bridge_test PASSED in 18.4s //tensorflow/compiler/tests:xla_ops_test_cpu PASSED in 49.8s //tensorflow/compiler/tests:xla_ops_test_cpu_mlir_bridge_test PASSED in 42.7s //tensorflow/compiler/tests:xla_test_test PASSED in 6.7s //tensorflow/compiler/tf2xla:const_analysis_test PASSED in 7.4s //tensorflow/compiler/tf2xla:cpu_function_runtime_test PASSED in 0.2s //tensorflow/compiler/tf2xla:functionalize_cond_test PASSED in 0.8s //tensorflow/compiler/tf2xla:functionalize_control_flow_test PASSED in 1.0s //tensorflow/compiler/tf2xla:fused_batchnorm_reserve_space_test_cpu PASSED in 33.6s //tensorflow/compiler/tf2xla:graph_compiler_test PASSED in 9.6s //tensorflow/compiler/tf2xla:literal_util_test PASSED in 1.0s //tensorflow/compiler/tf2xla:resource_operation_table_test PASSED in 10.1s //tensorflow/compiler/tf2xla:resource_util_test_cpu PASSED in 3.8s //tensorflow/compiler/tf2xla:sharding_util_test PASSED in 0.7s //tensorflow/compiler/tf2xla:tf2xla_opset_test PASSED in 9.9s //tensorflow/compiler/tf2xla:tf2xla_test PASSED in 24.7s //tensorflow/compiler/tf2xla:tf2xla_util_test PASSED in 1.9s //tensorflow/compiler/tf2xla:xla_compiler_test PASSED in 23.8s //tensorflow/compiler/tf2xla:xla_jit_compiled_cpu_function_test PASSED in 24.5s //tensorflow/compiler/tf2xla:xla_op_registry_test PASSED in 8.4s //tensorflow/compiler/tf2xla/kernels:rng_converter_utils_test PASSED in 2.7s //tensorflow/compiler/xla:array2d_test PASSED in 0.2s //tensorflow/compiler/xla:array3d_test PASSED in 0.7s //tensorflow/compiler/xla:array4d_test PASSED in 0.4s //tensorflow/compiler/xla:array_test PASSED in 0.5s //tensorflow/compiler/xla:bit_cast_test PASSED in 0.5s //tensorflow/compiler/xla:comparison_util_test PASSED in 0.2s //tensorflow/compiler/xla:debug_options_parsers_test PASSED in 0.6s //tensorflow/compiler/xla:index_util_test PASSED in 0.8s //tensorflow/compiler/xla:iterator_util_test PASSED in 0.2s //tensorflow/compiler/xla:layout_test PASSED in 0.1s //tensorflow/compiler/xla:layout_util_test PASSED in 0.1s //tensorflow/compiler/xla:literal_test PASSED in 1.5s //tensorflow/compiler/xla:parse_flags_from_env_test PASSED in 0.9s //tensorflow/compiler/xla:permutation_util_test PASSED in 0.2s //tensorflow/compiler/xla:primitive_util_test PASSED in 0.1s //tensorflow/compiler/xla:refcounting_hash_map_test PASSED in 0.4s //tensorflow/compiler/xla:reference_util_test PASSED in 0.6s //tensorflow/compiler/xla:shape_test PASSED in 0.3s //tensorflow/compiler/xla:shape_tree_test PASSED in 0.1s //tensorflow/compiler/xla:shape_util_test PASSED in 2.0s //tensorflow/compiler/xla:status_macros_test PASSED in 0.8s //tensorflow/compiler/xla:text_literal_reader_test PASSED in 0.2s //tensorflow/compiler/xla:text_literal_writer_test PASSED in 0.5s //tensorflow/compiler/xla:types_test PASSED in 0.5s //tensorflow/compiler/xla:util_test PASSED in 0.2s //tensorflow/compiler/xla:window_util_test PASSED in 0.3s //tensorflow/compiler/xla/client:padding_test PASSED in 1.3s //tensorflow/compiler/xla/client:xla_builder_test PASSED in 0.3s //tensorflow/compiler/xla/client/lib:arithmetic_test_cpu PASSED in 10.9s //tensorflow/compiler/xla/client/lib:comparators_test_cpu PASSED in 14.6s //tensorflow/compiler/xla/client/lib:constants_test_cpu PASSED in 11.0s //tensorflow/compiler/xla/client/lib:logdet_test_cpu PASSED in 17.6s //tensorflow/compiler/xla/client/lib:math_test_cpu PASSED in 19.7s //tensorflow/compiler/xla/client/lib:matrix_test_cpu PASSED in 13.8s //tensorflow/compiler/xla/client/lib:pooling_test_cpu PASSED in 12.0s //tensorflow/compiler/xla/client/lib:qr_test_cpu PASSED in 15.3s //tensorflow/compiler/xla/client/lib:slicing_test_cpu PASSED in 13.2s //tensorflow/compiler/xla/client/lib:sorting_test_cpu PASSED in 11.7s //tensorflow/compiler/xla/examples/axpy:stablehlo_compile_test PASSED in 20.0s //tensorflow/compiler/xla/experiments/sm_bandwidth_benchmark:sm_bw_test PASSED in 0.1s //tensorflow/compiler/xla/hlo/evaluator:hlo_evaluator_test PASSED in 4.6s //tensorflow/compiler/xla/hlo/experimental/auto_sharding:auto_sharding_test PASSED in 9.1s //tensorflow/compiler/xla/hlo/transforms:hlo_constant_splitter_test PASSED in 1.1s //tensorflow/compiler/xla/hlo/utils:hlo_live_range_test PASSED in 1.1s //tensorflow/compiler/xla/hlo/utils:hlo_matchers_test PASSED in 2.9s //tensorflow/compiler/xla/hlo/utils:hlo_sharding_util_test PASSED in 0.2s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:collective_ops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:fft.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:legalize_i1_vector_transfers.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:library_ops_to_cpu_runtime.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:lmhlo_custom_call.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:remove_copies_to_out_params.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:rng_bit_generator.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_abi_legalization.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_infeed.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_memref_element_cast_to_llvm.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_outfeed.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_concurrent_regions.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_hlo_trace.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_launch.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memcpy.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memset.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_case.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_custom_call.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_fft.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cholesky.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_conv.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cublas_lt_matmul.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_gemm.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_infeed.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_outfeed.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_send_recv.mlir.test PASSED in 2.0s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_while.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:memref_get_global_to_arg.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:outline_cuda_graphs.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/framework/tests:legalize-xla-framework.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir/framework/tests:outline-with-xla-framework.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/framework/tests:xla-framework.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/math/transforms/tests:math_optimization.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/memref/transforms/tests:aligned_allocations.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops_verify.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir/runtime/ir/tests:testlib.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/runtime/transforms:calling_convention_test PASSED in 0.4s //tensorflow/compiler/xla/mlir/runtime/transforms:type_converter_test PASSED in 0.7s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:compilation_pipeline.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_asserts.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_custom_calls.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:export_functions.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:ordinal_assignment.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:rt_to_llvm.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:erase-op-without-results.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:inline-scf-while.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:reduce-scf-forall-bounds.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-constant.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-value.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-operand-with-constant.mlir.test PASSED in 1.7s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:return-operands-of-terminator-operands.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:truncate-function.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:bisect.mlir.test PASSED in 2.2s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:no-bug.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:snapshot.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/tools/mlir_replay/public:execution_trace_utils_test PASSED in 0.6s //tensorflow/compiler/xla/mlir/utils:error_util_test PASSED in 0.1s //tensorflow/compiler/xla/mlir/xla_cpu/tests:bufferize.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/xla_cpu/tests:invalid.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/xla_cpu/tests:ops.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/bufferization/hlo_one_shot_bufferize.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_hlo_broadcasts.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_hlo_no_broadcasts.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_mhlo.mlir.test PASSED in 1.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/sparse_chlo_legalize_to_linalg.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/buffer_reuse.mlir.test PASSED in 1.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/convert_deallocation_ops_to_llvm.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocate.mlir.test PASSED in 2.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocate_invalid.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_simplification.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_to_scf.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/split_alloc_tensors.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/add_debug_info.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/bufferization.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/collapse-shape.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/collect_stats.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/compose_extract_insert_slice.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/batch_matmul.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/conv_2d_nhwc_hwcf.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/dot.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/duplicate_fusions.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fibonacci.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fusion_outlining.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fusion_planning_for_cpu.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/inline_fusion_clusters.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_bcast_map.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_matmul.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_reduce_map.mlir.test PASSED in 1.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_reshape_map.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/matmul.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_1d.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_1d_map.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_2d.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_window.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reverse.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/scatter.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/sort.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/transpose.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/greedy_fusion.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/invalid.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/lower_vectors.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/nested_tiling_softmax.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/ops.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/optimize_linalg_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/rewrite_forall_to_for.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/simplify_dead_copy.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/tile_by_one.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/tiling_softmax.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/vectorize_copy.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/vectorize_for_cpu.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-select-and-scatter.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-affine.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-gpu.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-parallel-loops.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-tensor-op.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/ops.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo_gpu/lhlo_gpu_ops.mlir.test PASSED in 1.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/attrs.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/broadcast_propagation.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/bitcast.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/canonicalize.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/concatenate.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/convert.mlir.test PASSED in 1.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/convolution.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/custom_call.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/folder_limit.mlir.test PASSED in 1.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reduce.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reshape.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reverse.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/scatter.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/transpose.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/tuple.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/while.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/constraint_fusion.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/convert_to_signless.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/expand_hlo_tuples.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/expand_ops_simplifier.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/group_reduction_dimensions.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-collapse-elementwise-map.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-einsum-to-dot-general.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-gather-to-torch-index-select.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-rng-to-linalg.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-shape-ops-to-standard.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-sort.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-arithmetic.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo-only-dynamic.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo-unranked.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-linalg.mlir.test PASSED in 3.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-memref-unranked.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-memref.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-stablehlo-experimental.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-stablehlo.mlir.test PASSED in 1.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/inlining.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/invalid.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-control-flow.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-hlo-shape-computations.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-mhlo-to-thlo.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-to-std.mlir.test PASSED in 1.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/lower-complex.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/lower-general-dot.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/materialize-broadcasts.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/merge_assuming_ops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_bytecode_customizations.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_dot.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_gather.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_reduction.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_scatter.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_flatten_tuple.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_infer_shape_type_methods.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_ops_prettyprint.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_reduce_pretty_print.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/ops.mlir.test PASSED in 3.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/optimize-hlo.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/prepare-for-export.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/reify-result-types.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/restrict_max_rank.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/shape_legalize_to_hlo.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/shape_reification.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sink-constants-to-control-flow.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_gendot_lower.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_lower.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_ops.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_rewriting.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_transpose.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/stablehlo-legalize-to-hlo.mlir.test PASSED in 1.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/symbolic-shape-optimization.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/unfuse_batch_norm.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_bounds.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_conv_op.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_reduce_op.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_reduce_window_op.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_scatter_op.mlir.test PASSED in 1.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_select_and_scatter_op.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_while_op.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/while_prettyprint.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/bufferize.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/canonicalize.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/invalid.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/legalize_sort.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/ops.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/tiling.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:alloc_to_arg.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:assuming-structural-propagation.mlir.test PASSED in 0.3s //tensorflow/compiler/xla/mlir_hlo/tests:buffer_packing.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:bufferize.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:bufferize_one_shot.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:collapse_parallel_loops_to_1d_pass.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:detensorize_scf_ops.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:index_type_llvm_lowering.mlir.test PASSED in 1.7s //tensorflow/compiler/xla/mlir_hlo/tests:legalize-trigonometric-to-approximation.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:lower_index_cast.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:propagate_static_shapes.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:rank-specialization.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:scalarization.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:shape-component-analysis.mlir.test PASSED in 1.6s //tensorflow/compiler/xla/mlir_hlo/tests:shape_simplification.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:test_userange.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:tile_loops.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:unbufferize.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:unroll-loops.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tools/mlir_interpreter/framework/tests:interpreter_value_test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tools/mlir_interpreter/framework/tests:tensor_or_memref_test PASSED in 0.1s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:binary.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:nullary.mlir.test PASSED in 1.6s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:prepare-mhlo.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:ternary.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:unary.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/pjrt:host_callback_test PASSED in 0.4s //tensorflow/compiler/xla/pjrt:lru_cache_test PASSED in 0.1s //tensorflow/compiler/xla/pjrt:pjrt_api_test PASSED in 0.6s //tensorflow/compiler/xla/pjrt:pjrt_client_test_cpu PASSED in 13.2s //tensorflow/compiler/xla/pjrt:pjrt_compiler_test PASSED in 0.6s //tensorflow/compiler/xla/pjrt:pjrt_executable_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:pjrt_stream_executor_client_test PASSED in 15.4s //tensorflow/compiler/xla/pjrt:semaphore_test PASSED in 0.4s //tensorflow/compiler/xla/pjrt:tf_pjrt_client_test PASSED in 11.1s //tensorflow/compiler/xla/pjrt:tfrt_cpu_pjrt_client_test PASSED in 10.1s //tensorflow/compiler/xla/pjrt:tracked_device_buffer_test PASSED in 12.0s //tensorflow/compiler/xla/pjrt:tracked_tfrt_cpu_device_buffer_test PASSED in 0.6s //tensorflow/compiler/xla/pjrt:transpose_test PASSED in 61.2s //tensorflow/compiler/xla/pjrt/c:pjrt_c_api_cpu_test PASSED in 11.8s //tensorflow/compiler/xla/pjrt/c:pjrt_c_api_helpers_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt/distributed:client_server_test PASSED in 43.3s //tensorflow/compiler/xla/pjrt/distributed:service_test PASSED in 10.9s //tensorflow/compiler/xla/pjrt/gpu:se_gpu_pjrt_client_test PASSED in 3.9s //tensorflow/compiler/xla/python:outfeed_receiver_test_cpu PASSED in 16.8s //tensorflow/compiler/xla/python/ifrt:array_test PASSED in 0.4s //tensorflow/compiler/xla/python/ifrt:array_test_no_impl PASSED in 0.2s //tensorflow/compiler/xla/python/ifrt:client_test_no_impl PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:future_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:index_domain_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:index_test PASSED in 0.7s //tensorflow/compiler/xla/python/ifrt:shape_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:sharding_test PASSED in 0.2s //tensorflow/compiler/xla/python/ifrt:tuple_test_no_impl PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_array.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_assemble.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_attrs.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call_loaded_executable.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_disassemble.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_loaded_executable.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_reshard.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/python/ifrt/support:sharding_param_to_op_sharding_test PASSED in 0.5s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_array_impl_test_tfrt_cpu PASSED in 19.3s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_client_impl_test_tfrt_cpu PASSED in 19.3s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_executable_impl_test_tfrt_cpu PASSED in 14.3s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_tuple_impl_test_tfrt_cpu PASSED in 10.4s //tensorflow/compiler/xla/python/pjrt_ifrt:xla_executable_test_no_impl PASSED in 2.2s //tensorflow/compiler/xla/python_api:xla_literal_test PASSED in 0.9s //tensorflow/compiler/xla/python_api:xla_shape_test PASSED in 1.0s //tensorflow/compiler/xla/rpc:grpc_client_test PASSED in 4.0s //tensorflow/compiler/xla/runtime:arguments_test PASSED in 0.6s //tensorflow/compiler/xla/runtime:async_runtime_test PASSED in 1.1s //tensorflow/compiler/xla/runtime:custom_call_test PASSED in 2.4s //tensorflow/compiler/xla/runtime:diagnostics_test PASSED in 0.4s //tensorflow/compiler/xla/runtime:executable_test PASSED in 3.1s //tensorflow/compiler/xla/runtime:ffi_test PASSED in 2.9s //tensorflow/compiler/xla/runtime:map_by_type_test PASSED in 0.3s //tensorflow/compiler/xla/runtime:module_test PASSED in 0.4s //tensorflow/compiler/xla/runtime:results_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:state_test PASSED in 0.4s //tensorflow/compiler/xla/runtime:symbolic_shape_test PASSED in 0.7s //tensorflow/compiler/xla/runtime:type_id_test PASSED in 0.2s //tensorflow/compiler/xla/service:algebraic_simplifier_overflow_test_cpu PASSED in 10.3s //tensorflow/compiler/xla/service:algebraic_simplifier_test PASSED in 7.6s //tensorflow/compiler/xla/service:all_gather_broadcast_reorder_test PASSED in 1.4s //tensorflow/compiler/xla/service:all_gather_combiner_test PASSED in 2.8s //tensorflow/compiler/xla/service:all_gather_decomposer_test PASSED in 1.9s //tensorflow/compiler/xla/service:all_reduce_combiner_test PASSED in 1.7s //tensorflow/compiler/xla/service:all_reduce_contiguous_test PASSED in 1.7s //tensorflow/compiler/xla/service:all_reduce_folder_test PASSED in 1.2s //tensorflow/compiler/xla/service:all_reduce_promotion_test PASSED in 0.9s //tensorflow/compiler/xla/service:all_reduce_reassociate_test PASSED in 1.2s //tensorflow/compiler/xla/service:all_reduce_simplifier_test PASSED in 4.0s //tensorflow/compiler/xla/service:ar_crs_combiner_test PASSED in 1.0s //tensorflow/compiler/xla/service:async_collective_creator_test PASSED in 1.4s //tensorflow/compiler/xla/service:async_op_canonicalizer_test PASSED in 2.0s //tensorflow/compiler/xla/service:batch_dot_simplification_test PASSED in 1.0s //tensorflow/compiler/xla/service:batchnorm_expander_test_cpu PASSED in 14.5s //tensorflow/compiler/xla/service:bfloat16_conversion_folding_test PASSED in 2.3s //tensorflow/compiler/xla/service:bfloat16_propagation_test PASSED in 3.0s //tensorflow/compiler/xla/service:bitcast_dtypes_expander_test PASSED in 1.3s //tensorflow/compiler/xla/service:broadcast_canonicalizer_test PASSED in 1.8s //tensorflow/compiler/xla/service:buffer_assignment_test PASSED in 15.9s //tensorflow/compiler/xla/service:call_graph_test PASSED in 1.2s //tensorflow/compiler/xla/service:call_inliner_test PASSED in 1.6s //tensorflow/compiler/xla/service:change_op_data_type_test PASSED in 1.1s //tensorflow/compiler/xla/service:collective_ops_utils_test PASSED in 0.7s //tensorflow/compiler/xla/service:collective_transformation_reorderer_test PASSED in 1.6s //tensorflow/compiler/xla/service:collectives_schedule_linearizer_test PASSED in 5.8s //tensorflow/compiler/xla/service:compilation_environments_test PASSED in 0.1s //tensorflow/compiler/xla/service:conditional_canonicalizer_test PASSED in 1.4s //tensorflow/compiler/xla/service:conditional_code_motion_test PASSED in 3.4s //tensorflow/compiler/xla/service:conditional_simplifier_test PASSED in 3.0s //tensorflow/compiler/xla/service:conditional_to_select_test PASSED in 1.5s //tensorflow/compiler/xla/service:convert_async_collectives_to_sync_test PASSED in 3.4s //tensorflow/compiler/xla/service:convert_mover_test PASSED in 1.3s //tensorflow/compiler/xla/service:convert_operand_folding_test PASSED in 1.7s //tensorflow/compiler/xla/service:convolution_4d_expander_test PASSED in 1.1s //tensorflow/compiler/xla/service:convolution_group_converter_test PASSED in 1.5s //tensorflow/compiler/xla/service:convolution_pred_expander_test PASSED in 1.6s //tensorflow/compiler/xla/service:copy_insertion_test PASSED in 3.2s //tensorflow/compiler/xla/service:custom_call_status_test PASSED in 0.2s //tensorflow/compiler/xla/service:defuser_test PASSED in 1.5s //tensorflow/compiler/xla/service:despecializer_test PASSED in 4.1s //tensorflow/compiler/xla/service:dfs_hlo_visitor_with_default_test PASSED in 1.2s //tensorflow/compiler/xla/service:dot_decomposer_test PASSED in 1.7s //tensorflow/compiler/xla/service:dot_dimension_merger_test PASSED in 1.3s //tensorflow/compiler/xla/service:dot_merger_test PASSED in 1.3s //tensorflow/compiler/xla/service:dynamic_dimension_inference_test PASSED in 1.9s //tensorflow/compiler/xla/service:dynamic_dimension_simplifier_test PASSED in 1.5s //tensorflow/compiler/xla/service:dynamic_index_splitter_test PASSED in 1.7s //tensorflow/compiler/xla/service:dynamic_padder_test_cpu PASSED in 28.0s //tensorflow/compiler/xla/service:dynamic_parameter_binding_test PASSED in 1.7s //tensorflow/compiler/xla/service:dynamic_update_slice_test_cpu PASSED in 16.4s //tensorflow/compiler/xla/service:elemental_ir_emitter_test_cpu PASSED in 19.1s //tensorflow/compiler/xla/service:flatten_call_graph_test PASSED in 1.8s //tensorflow/compiler/xla/service:float_normalization_test PASSED in 1.0s //tensorflow/compiler/xla/service:fusion_node_indexing_evaluation_test PASSED in 2.1s //tensorflow/compiler/xla/service:gather_expander_test PASSED in 1.5s //tensorflow/compiler/xla/service:gather_simplifier_test PASSED in 2.4s //tensorflow/compiler/xla/service:heap_simulator_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_alias_analysis_test PASSED in 1.9s //tensorflow/compiler/xla/service:hlo_casting_utils_test PASSED in 9.7s //tensorflow/compiler/xla/service:hlo_computation_deduplicator_test PASSED in 1.6s //tensorflow/compiler/xla/service:hlo_computation_test PASSED in 5.5s //tensorflow/compiler/xla/service:hlo_constant_folding_test PASSED in 7.1s //tensorflow/compiler/xla/service:hlo_cost_analysis_test PASSED in 12.7s //tensorflow/compiler/xla/service:hlo_creation_utils_test PASSED in 4.4s //tensorflow/compiler/xla/service:hlo_cse_test PASSED in 18.2s //tensorflow/compiler/xla/service:hlo_dataflow_analysis_test PASSED in 2.1s //tensorflow/compiler/xla/service:hlo_dce_test PASSED in 2.2s //tensorflow/compiler/xla/service:hlo_domain_test PASSED in 1.9s //tensorflow/compiler/xla/service:hlo_element_type_converter_test PASSED in 1.3s //tensorflow/compiler/xla/service:hlo_execution_profile_test PASSED in 10.1s //tensorflow/compiler/xla/service:hlo_graph_dumper_test PASSED in 2.6s //tensorflow/compiler/xla/service:hlo_input_output_alias_config_test PASSED in 1.9s //tensorflow/compiler/xla/service:hlo_instruction_test PASSED in 3.1s //tensorflow/compiler/xla/service:hlo_liveness_analysis_test PASSED in 1.4s //tensorflow/compiler/xla/service:hlo_memory_scheduler_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_module_dce_test PASSED in 1.4s //tensorflow/compiler/xla/service:hlo_module_metadata_test PASSED in 0.4s //tensorflow/compiler/xla/service:hlo_module_test PASSED in 1.8s //tensorflow/compiler/xla/service:hlo_opcode_test PASSED in 0.4s //tensorflow/compiler/xla/service:hlo_ordering_test PASSED in 2.6s //tensorflow/compiler/xla/service:hlo_parser_test PASSED in 0.3s //tensorflow/compiler/xla/service:hlo_pass_pipeline_test PASSED in 1.3s //tensorflow/compiler/xla/service:hlo_phi_graph_test PASSED in 0.2s //tensorflow/compiler/xla/service:hlo_proto_util_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_reachability_test PASSED in 2.2s //tensorflow/compiler/xla/service:hlo_rematerialization_test PASSED in 2.6s //tensorflow/compiler/xla/service:hlo_rematerialization_test_utils_test PASSED in 1.7s //tensorflow/compiler/xla/service:hlo_replication_analysis_test PASSED in 1.2s //tensorflow/compiler/xla/service:hlo_schedule_test PASSED in 1.0s //tensorflow/compiler/xla/service:hlo_sharding_test PASSED in 1.5s //tensorflow/compiler/xla/service:hlo_value_semantics_analysis_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_verifier_test PASSED in 3.6s //tensorflow/compiler/xla/service:indexed_array_analysis_test PASSED in 1.7s //tensorflow/compiler/xla/service:instruction_fusion_test PASSED in 1.3s //tensorflow/compiler/xla/service:latency_hiding_scheduler_test PASSED in 3.0s //tensorflow/compiler/xla/service:layout_assignment_test PASSED in 6.2s //tensorflow/compiler/xla/service:layout_normalization_test PASSED in 4.3s //tensorflow/compiler/xla/service:logistic_expander_test PASSED in 2.6s //tensorflow/compiler/xla/service:loop_schedule_linearizer_test PASSED in 1.5s //tensorflow/compiler/xla/service:map_inliner_test PASSED in 1.7s //tensorflow/compiler/xla/service:mapped_ptr_container_sorter_test PASSED in 0.2s //tensorflow/compiler/xla/service:memory_space_assignment_best_fit_repacker_test PASSED in 0.9s //tensorflow/compiler/xla/service:memory_space_assignment_test PASSED in 3.6s //tensorflow/compiler/xla/service:memory_space_propagation_test PASSED in 0.9s //tensorflow/compiler/xla/service:name_uniquer_test PASSED in 0.2s //tensorflow/compiler/xla/service:operand_upcaster_test PASSED in 1.3s //tensorflow/compiler/xla/service:optimize_input_output_buffer_alias_test PASSED in 1.1s //tensorflow/compiler/xla/service:pattern_matcher_gmock_test PASSED in 0.8s //tensorflow/compiler/xla/service:pattern_matcher_test PASSED in 3.3s //tensorflow/compiler/xla/service:profile_guided_latency_estimator_test PASSED in 2.6s //tensorflow/compiler/xla/service:real_imag_expander_test PASSED in 1.0s //tensorflow/compiler/xla/service:reduce_decomposer_test PASSED in 1.4s //tensorflow/compiler/xla/service:reduce_scatter_combiner_test PASSED in 1.7s //tensorflow/compiler/xla/service:reduce_scatter_decomposer_test PASSED in 1.7s //tensorflow/compiler/xla/service:reduce_scatter_reassociate_test PASSED in 3.2s //tensorflow/compiler/xla/service:reshape_decomposer_test PASSED in 2.2s //tensorflow/compiler/xla/service:reshape_mover_test PASSED in 2.7s //tensorflow/compiler/xla/service:result_caster_test PASSED in 1.5s //tensorflow/compiler/xla/service:root_instruction_sinker_test PASSED in 1.1s //tensorflow/compiler/xla/service:scatter_expander_test PASSED in 1.6s //tensorflow/compiler/xla/service:scatter_simplifier_test PASSED in 1.7s //tensorflow/compiler/xla/service:select_and_scatter_expander_test PASSED in 3.0s //tensorflow/compiler/xla/service:shape_inference_test PASSED in 0.5s //tensorflow/compiler/xla/service:shaped_buffer_test PASSED in 9.9s //tensorflow/compiler/xla/service:sharding_propagation_test PASSED in 14.8s //tensorflow/compiler/xla/service:sharding_remover_test PASSED in 1.6s //tensorflow/compiler/xla/service:simplify_fp_conversions_test PASSED in 1.4s //tensorflow/compiler/xla/service:slice_sinker_test PASSED in 1.8s //tensorflow/compiler/xla/service:sort_simplifier_test PASSED in 2.4s //tensorflow/compiler/xla/service:space_to_batch_converter_test PASSED in 1.9s //tensorflow/compiler/xla/service:stable_sort_expander_test PASSED in 1.2s //tensorflow/compiler/xla/service:stochastic_convert_decomposer_test PASSED in 2.0s //tensorflow/compiler/xla/service:stream_pool_test PASSED in 0.5s //tensorflow/compiler/xla/service:topk_rewriter_test PASSED in 6.7s //tensorflow/compiler/xla/service:transpose_folding_test PASSED in 2.1s //tensorflow/compiler/xla/service:tuple_points_to_analysis_test PASSED in 1.1s //tensorflow/compiler/xla/service:tuple_simplifier_test PASSED in 2.5s //tensorflow/compiler/xla/service:tuple_util_test PASSED in 1.4s //tensorflow/compiler/xla/service:while_loop_all_reduce_code_motion_test PASSED in 2.1s //tensorflow/compiler/xla/service:while_loop_analysis_test PASSED in 1.9s //tensorflow/compiler/xla/service:while_loop_concat_code_motion_test PASSED in 1.5s //tensorflow/compiler/xla/service:while_loop_constant_sinking_test PASSED in 1.4s //tensorflow/compiler/xla/service:while_loop_expensive_invariant_code_motion_test PASSED in 1.8s //tensorflow/compiler/xla/service:while_loop_invariant_code_motion_test PASSED in 1.2s //tensorflow/compiler/xla/service:while_loop_simplifier_test PASSED in 3.8s //tensorflow/compiler/xla/service:while_loop_trip_count_annotator_test PASSED in 2.9s //tensorflow/compiler/xla/service:while_util_test PASSED in 1.3s //tensorflow/compiler/xla/service:xla_aot_compile_stablehlo_cpu_test PASSED in 14.8s //tensorflow/compiler/xla/service:xla_debug_info_manager_test PASSED in 0.9s //tensorflow/compiler/xla/service:zero_sized_hlo_elimination_test PASSED in 1.0s //tensorflow/compiler/xla/service/cpu:conv_canonicalization_test PASSED in 1.9s //tensorflow/compiler/xla/service/cpu:cpu_eigen_tensor_alignment_test PASSED in 1.6s //tensorflow/compiler/xla/service/cpu:cpu_instruction_fusion_test PASSED in 2.3s //tensorflow/compiler/xla/service/cpu:cpu_layout_assignment_test PASSED in 4.6s //tensorflow/compiler/xla/service/cpu:ir_emission_utils_test PASSED in 1.0s //tensorflow/compiler/xla/service/cpu:parallel_task_assignment_test PASSED in 3.7s //tensorflow/compiler/xla/service/cpu:runtime_fft_test PASSED in 0.1s //tensorflow/compiler/xla/service/cpu:shape_partition_test PASSED in 0.8s //tensorflow/compiler/xla/service/cpu:xfeed_manager_test PASSED in 0.9s //tensorflow/compiler/xla/service/cpu/tests:cpu_bytesizeof_test PASSED in 0.6s //tensorflow/compiler/xla/service/cpu/tests:cpu_dyn_shape_test PASSED in 10.6s //tensorflow/compiler/xla/service/cpu/tests:cpu_eigen_dot_operation_test PASSED in 14.2s //tensorflow/compiler/xla/service/cpu/tests:cpu_external_constants_test PASSED in 31.2s //tensorflow/compiler/xla/service/cpu/tests:cpu_fusion_test PASSED in 10.8s //tensorflow/compiler/xla/service/cpu/tests:cpu_infeed_test PASSED in 11.1s //tensorflow/compiler/xla/service/cpu/tests:cpu_intrinsic_test PASSED in 14.7s //tensorflow/compiler/xla/service/cpu/tests:cpu_key_value_sort_test PASSED in 11.8s //tensorflow/compiler/xla/service/cpu/tests:cpu_literal_caching_test PASSED in 9.9s //tensorflow/compiler/xla/service/cpu/tests:cpu_noalias_test PASSED in 10.5s //tensorflow/compiler/xla/service/cpu/tests:cpu_outfeed_test PASSED in 10.4s //tensorflow/compiler/xla/service/cpu/tests:cpu_profiling_test PASSED in 16.0s //tensorflow/compiler/xla/service/cpu/tests:cpu_spmd_compile_test PASSED in 11.2s //tensorflow/compiler/xla/service/cpu/tests:cpu_topk_test PASSED in 14.2s //tensorflow/compiler/xla/service/cpu/tests:cpu_vectorization_test PASSED in 18.3s //tensorflow/compiler/xla/service/cpu/tests:cpu_while_test PASSED in 17.5s //tensorflow/compiler/xla/service/cpu/tests:tree_reduction_rewriter_test PASSED in 12.5s //tensorflow/compiler/xla/service/gpu:alias_passthrough_params_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu:all_reduce_blueconnect_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu:backend_configs_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu:cublas_pad_for_gemms_test PASSED in 2.6s //tensorflow/compiler/xla/service/gpu:cudnn_pad_for_convolutions_test PASSED in 2.7s //tensorflow/compiler/xla/service/gpu:cudnn_simplify_padding_test PASSED in 2.7s //tensorflow/compiler/xla/service/gpu:cudnn_support_utils_test PASSED in 2.5s //tensorflow/compiler/xla/service/gpu:cudnn_vectorize_convolutions_test PASSED in 2.6s //tensorflow/compiler/xla/service/gpu:fusion_merger_test PASSED in 6.0s //tensorflow/compiler/xla/service/gpu:gemm_rewriter_triton_test PASSED in 3.5s //tensorflow/compiler/xla/service/gpu:gpu_async_collective_annotator_test PASSED in 1.0s //tensorflow/compiler/xla/service/gpu:gpu_conv_padding_legalization_test PASSED in 1.2s //tensorflow/compiler/xla/service/gpu:gpu_conv_rewriter_test PASSED in 2.0s //tensorflow/compiler/xla/service/gpu:gpu_convert_async_collectives_to_sync_test PASSED in 1.7s //tensorflow/compiler/xla/service/gpu:gpu_copy_insertion_test PASSED in 11.1s //tensorflow/compiler/xla/service/gpu:gpu_fusible_test PASSED in 2.4s //tensorflow/compiler/xla/service/gpu:gpu_hlo_cost_analysis_test PASSED in 2.3s //tensorflow/compiler/xla/service/gpu:gpu_performance_model_test PASSED in 3.9s //tensorflow/compiler/xla/service/gpu:gpu_sanitize_constant_names_test PASSED in 2.0s //tensorflow/compiler/xla/service/gpu:hlo_algorithm_denylist_test PASSED in 0.5s //tensorflow/compiler/xla/service/gpu:hlo_fusion_stats_test PASSED in 2.1s //tensorflow/compiler/xla/service/gpu:instruction_fusion_test PASSED in 3.1s //tensorflow/compiler/xla/service/gpu:ir_emission_utils_test PASSED in 2.2s //tensorflow/compiler/xla/service/gpu:matmul_utils_test PASSED in 13.3s //tensorflow/compiler/xla/service/gpu:move_copy_to_users_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu:multi_output_fusion_test PASSED in 6.6s //tensorflow/compiler/xla/service/gpu:non_atomically_upgradeable_rw_lock_test PASSED in 0.5s //tensorflow/compiler/xla/service/gpu:reduction_splitter_test PASSED in 2.3s //tensorflow/compiler/xla/service/gpu:scatter_slice_simplifier_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu:target_util_test PASSED in 0.8s //tensorflow/compiler/xla/service/gpu:topk_splitter_test PASSED in 40.0s //tensorflow/compiler/xla/service/gpu:variadic_op_splitter_test PASSED in 2.0s //tensorflow/compiler/xla/service/gpu:while_transformer_test PASSED in 2.9s //tensorflow/compiler/xla/service/gpu/llvm_gpu_backend:utils_test PASSED in 0.8s //tensorflow/compiler/xla/service/gpu/tests:gpu_reduce_scatter_creator_test PASSED in 2.3s //tensorflow/compiler/xla/service/gpu/tests:reduction_degenerate_dim_remover_test PASSED in 1.9s //tensorflow/compiler/xla/service/gpu/tests:reduction_dimension_grouper_test PASSED in 1.2s //tensorflow/compiler/xla/service/gpu/tests:tree_reduction_rewriter_test PASSED in 3.1s //tensorflow/compiler/xla/service/graphcycles:graphcycles_test PASSED in 0.7s //tensorflow/compiler/xla/service/graphcycles:ordered_set_test PASSED in 0.4s //tensorflow/compiler/xla/service/llvm_ir:alias_analysis_test PASSED in 11.0s //tensorflow/compiler/xla/service/llvm_ir:ir_array_test PASSED in 1.1s //tensorflow/compiler/xla/service/spmd:canonicalize_all_gather_for_cse_test PASSED in 2.7s //tensorflow/compiler/xla/service/spmd:collective_permute_motion_test PASSED in 1.0s //tensorflow/compiler/xla/service/spmd:partition_assignment_test PASSED in 1.3s //tensorflow/compiler/xla/service/spmd:schedule_aware_collective_ops_cse_test PASSED in 1.0s //tensorflow/compiler/xla/service/spmd:spmd_partitioner_test PASSED in 3.6s //tensorflow/compiler/xla/service/spmd:stateful_rng_spmd_partitioner_test PASSED in 2.0s //tensorflow/compiler/xla/stream_executor:dnn_test PASSED in 0.4s //tensorflow/compiler/xla/stream_executor:stream_test PASSED in 0.3s //tensorflow/compiler/xla/stream_executor/host:host_stream_test PASSED in 0.2s //tensorflow/compiler/xla/tests:all_reduce_test_cpu PASSED in 14.0s //tensorflow/compiler/xla/tests:axpy_simple_test_cpu PASSED in 15.7s //tensorflow/compiler/xla/tests:bad_rng_shape_validation_test_cpu PASSED in 13.1s //tensorflow/compiler/xla/tests:binop_scaling_test_cpu PASSED in 10.5s //tensorflow/compiler/xla/tests:bitcast_convert_test_cpu PASSED in 9.8s //tensorflow/compiler/xla/tests:broadcast_simple_test_cpu PASSED in 17.2s //tensorflow/compiler/xla/tests:broadcast_test_cpu PASSED in 13.5s //tensorflow/compiler/xla/tests:buffer_donation_test_cpu PASSED in 13.1s //tensorflow/compiler/xla/tests:call_test_cpu PASSED in 11.6s //tensorflow/compiler/xla/tests:check_execution_arity_test_cpu PASSED in 13.4s //tensorflow/compiler/xla/tests:cholesky_test_cpu PASSED in 20.5s //tensorflow/compiler/xla/tests:client_test_cpu PASSED in 12.8s //tensorflow/compiler/xla/tests:collective_ops_test_cpu PASSED in 37.1s //tensorflow/compiler/xla/tests:compilation_cache_test_cpu PASSED in 13.4s //tensorflow/compiler/xla/tests:compute_constant_test_cpu PASSED in 12.7s //tensorflow/compiler/xla/tests:concat_test_cpu PASSED in 16.1s //tensorflow/compiler/xla/tests:constant_reduction_function_test_cpu PASSED in 11.9s //tensorflow/compiler/xla/tests:constants_test_cpu PASSED in 12.0s //tensorflow/compiler/xla/tests:convert_test_cpu PASSED in 12.9s //tensorflow/compiler/xla/tests:copy_test_cpu PASSED in 13.6s //tensorflow/compiler/xla/tests:cpu_gpu_fusion_test_cpu PASSED in 15.2s //tensorflow/compiler/xla/tests:custom_call_test_cpu PASSED in 18.5s //tensorflow/compiler/xla/tests:deallocation_test_cpu PASSED in 11.4s //tensorflow/compiler/xla/tests:deconstruct_tuple_test_cpu PASSED in 11.9s //tensorflow/compiler/xla/tests:deep_graph_test_cpu PASSED in 17.9s //tensorflow/compiler/xla/tests:execution_profile_test_cpu PASSED in 10.0s //tensorflow/compiler/xla/tests:fft_test_cpu PASSED in 14.8s //tensorflow/compiler/xla/tests:float8_test_cpu PASSED in 17.2s //tensorflow/compiler/xla/tests:floor_ceil_test_cpu PASSED in 11.4s //tensorflow/compiler/xla/tests:fmax_fmin_test_cpu PASSED in 12.5s //tensorflow/compiler/xla/tests:gather_operation_test_cpu PASSED in 24.7s //tensorflow/compiler/xla/tests:get_dimension_size_test_cpu PASSED in 11.9s //tensorflow/compiler/xla/tests:half_test_cpu PASSED in 16.4s //tensorflow/compiler/xla/tests:hlo_metadata_test PASSED in 10.4s //tensorflow/compiler/xla/tests:literal_test_util_test PASSED in 7.2s //tensorflow/compiler/xla/tests:local_client_allocation_test_cpu PASSED in 12.7s //tensorflow/compiler/xla/tests:local_client_aot_test PASSED in 0.5s //tensorflow/compiler/xla/tests:log_test_cpu PASSED in 14.0s //tensorflow/compiler/xla/tests:map_test_cpu PASSED in 12.1s //tensorflow/compiler/xla/tests:matrix_ops_simple_test_cpu PASSED in 29.8s //tensorflow/compiler/xla/tests:multidimensional_slice_test_cpu PASSED in 11.6s //tensorflow/compiler/xla/tests:multiple_devices_on_host_test PASSED in 13.4s //tensorflow/compiler/xla/tests:multithreaded_compilation_test_cpu PASSED in 13.2s //tensorflow/compiler/xla/tests:outfeed_in_nested_computation_test_cpu PASSED in 19.0s //tensorflow/compiler/xla/tests:pad_test_cpu PASSED in 13.5s //tensorflow/compiler/xla/tests:pred_test_cpu PASSED in 11.2s //tensorflow/compiler/xla/tests:query_inferred_shape_test_cpu PASSED in 15.2s //tensorflow/compiler/xla/tests:reduce_hlo_test_cpu PASSED in 12.1s //tensorflow/compiler/xla/tests:reduce_precision_test_cpu PASSED in 13.2s //tensorflow/compiler/xla/tests:replay_test_cpu PASSED in 14.9s //tensorflow/compiler/xla/tests:reshape_motion_test_cpu PASSED in 17.8s //tensorflow/compiler/xla/tests:reverse_test_cpu PASSED in 13.7s //tensorflow/compiler/xla/tests:round_trip_packed_literal_test_cpu PASSED in 11.9s //tensorflow/compiler/xla/tests:round_trip_transfer_test_cpu PASSED in 11.6s //tensorflow/compiler/xla/tests:sample_text_test_cpu PASSED in 14.2s //tensorflow/compiler/xla/tests:scatter_test_cpu PASSED in 17.1s //tensorflow/compiler/xla/tests:select_test_cpu PASSED in 16.8s //tensorflow/compiler/xla/tests:test_utils_test_cpu PASSED in 16.4s //tensorflow/compiler/xla/tests:token_hlo_test_cpu PASSED in 13.5s //tensorflow/compiler/xla/tests:topk_test_cpu PASSED in 12.4s //tensorflow/compiler/xla/tests:transfer_manager_test_cpu PASSED in 20.3s //tensorflow/compiler/xla/tests:transpose_test_cpu PASSED in 13.4s //tensorflow/compiler/xla/tests:tuple_test_cpu PASSED in 13.3s //tensorflow/compiler/xla/tests:unary_op_test_cpu PASSED in 13.7s //tensorflow/compiler/xla/tests:value_inference_test_cpu PASSED in 10.0s //tensorflow/compiler/xla/tests:vector_ops_reduce_test_cpu PASSED in 13.1s //tensorflow/compiler/xla/tests:vector_ops_simple_test_cpu PASSED in 12.3s //tensorflow/compiler/xla/tests:while_test_cpu PASSED in 14.5s //tensorflow/compiler/xla/tests/fuzz:rand_000000_cpu PASSED in 12.3s //tensorflow/compiler/xla/tests/fuzz:rand_000003_cpu PASSED in 12.8s //tensorflow/compiler/xla/tests/fuzz:rand_000005_cpu PASSED in 13.6s //tensorflow/compiler/xla/tests/fuzz:rand_000006_cpu PASSED in 13.1s //tensorflow/compiler/xla/tests/fuzz:rand_000007_cpu PASSED in 15.8s //tensorflow/compiler/xla/tests/fuzz:rand_000008_cpu PASSED in 10.8s //tensorflow/compiler/xla/tests/fuzz:rand_000009_cpu PASSED in 12.5s //tensorflow/compiler/xla/tests/fuzz:rand_000013_cpu PASSED in 12.6s //tensorflow/compiler/xla/tests/fuzz:rand_000015_cpu PASSED in 13.4s //tensorflow/compiler/xla/tests/fuzz:rand_000016_cpu PASSED in 10.9s //tensorflow/compiler/xla/tests/fuzz:rand_000017_cpu PASSED in 14.5s //tensorflow/compiler/xla/tests/fuzz:rand_000018_cpu PASSED in 15.3s //tensorflow/compiler/xla/tests/fuzz:rand_000019_cpu PASSED in 10.8s //tensorflow/compiler/xla/tests/fuzz:rand_000020_cpu PASSED in 12.6s //tensorflow/compiler/xla/tests/fuzz:rand_000022_cpu PASSED in 12.2s //tensorflow/compiler/xla/tests/fuzz:rand_000024_cpu PASSED in 11.4s //tensorflow/compiler/xla/tests/fuzz:rand_000025_cpu PASSED in 14.5s //tensorflow/compiler/xla/tests/fuzz:rand_000026_cpu PASSED in 10.5s //tensorflow/compiler/xla/tests/fuzz:rand_000030_cpu PASSED in 10.4s //tensorflow/compiler/xla/tests/fuzz:rand_000031_cpu PASSED in 11.4s //tensorflow/compiler/xla/tests/fuzz:rand_000032_cpu PASSED in 12.4s //tensorflow/compiler/xla/tests/fuzz:rand_000033_cpu PASSED in 14.2s //tensorflow/compiler/xla/tests/fuzz:rand_000034_cpu PASSED in 15.7s //tensorflow/compiler/xla/tests/fuzz:rand_000035_cpu PASSED in 14.9s //tensorflow/compiler/xla/tests/fuzz:rand_000036_cpu PASSED in 15.5s //tensorflow/compiler/xla/tests/fuzz:rand_000039_cpu PASSED in 12.3s //tensorflow/compiler/xla/tests/fuzz:rand_000040_cpu PASSED in 13.0s //tensorflow/compiler/xla/tests/fuzz:rand_000041_cpu PASSED in 11.7s //tensorflow/compiler/xla/tests/fuzz:rand_000043_cpu PASSED in 12.0s //tensorflow/compiler/xla/tests/fuzz:rand_000049_cpu PASSED in 12.2s //tensorflow/compiler/xla/tests/fuzz:rand_000053_cpu PASSED in 13.4s //tensorflow/compiler/xla/tests/fuzz:rand_000056_cpu PASSED in 12.3s //tensorflow/compiler/xla/tests/fuzz:rand_000059_cpu PASSED in 11.9s //tensorflow/compiler/xla/tests/fuzz:rand_000061_cpu PASSED in 12.8s //tensorflow/compiler/xla/tests/fuzz:rand_000062_cpu PASSED in 12.8s //tensorflow/compiler/xla/tests/fuzz:rand_000064_cpu PASSED in 14.4s //tensorflow/compiler/xla/tests/fuzz:rand_000066_cpu PASSED in 15.7s //tensorflow/compiler/xla/tests/fuzz:rand_000069_cpu PASSED in 12.4s //tensorflow/compiler/xla/tests/fuzz:rand_000071_cpu PASSED in 11.4s //tensorflow/compiler/xla/tests/fuzz:rand_000077_cpu PASSED in 12.9s //tensorflow/compiler/xla/tests/fuzz:rand_000078_cpu PASSED in 12.7s //tensorflow/compiler/xla/tests/fuzz:rand_000079_cpu PASSED in 12.0s //tensorflow/compiler/xla/tests/fuzz:rand_000081_cpu PASSED in 13.0s //tensorflow/compiler/xla/tests/fuzz:rand_000084_cpu PASSED in 10.2s //tensorflow/compiler/xla/tests/fuzz:rand_000085_cpu PASSED in 13.6s //tensorflow/compiler/xla/tests/fuzz:rand_000086_cpu PASSED in 15.2s //tensorflow/compiler/xla/tests/fuzz:rand_000088_cpu PASSED in 10.6s //tensorflow/compiler/xla/tests/fuzz:rand_000089_cpu PASSED in 10.5s //tensorflow/compiler/xla/tests/fuzz:rand_000090_cpu PASSED in 14.3s //tensorflow/compiler/xla/tests/fuzz:rand_000092_cpu PASSED in 11.7s //tensorflow/compiler/xla/tests/fuzz:rand_000094_cpu PASSED in 16.8s //tensorflow/compiler/xla/tests/fuzz:rand_000095_cpu PASSED in 13.6s //tensorflow/compiler/xla/tools:hlo_control_flow_flattening_test PASSED in 2.2s //tensorflow/compiler/xla/tools:hlo_extractor_test PASSED in 2.2s //tensorflow/compiler/xla/tools:hlo_module_loader_test PASSED in 1.5s //tensorflow/compiler/xla/tools:interactive_graphviz_bin_test PASSED in 0.9s //tensorflow/compiler/xla/tools:run_hlo_module_bin_test PASSED in 1.2s //tensorflow/compiler/xla/tools/hlo_bisect:hlo_bisect_state_test PASSED in 1.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo:hlo_utils_test PASSED in 1.1s //tensorflow/compiler/xla/translate/hlo_to_mhlo:mlir_hlo_builder_test PASSED in 1.6s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:bool_compare.hlotxt.test PASSED in 1.4s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:case_conditional.hlotxt.test PASSED in 2.8s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:dynamic_param.hlo.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:entry_computation_layout.hlotxt.test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:frontend_attributes.hlotxt.test PASSED in 0.8s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fully_connected_reference_model.hlotxt.test PASSED in 0.8s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fusion.hlotxt.test PASSED in 0.6s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:if_conditional.hlotxt.test PASSED in 1.1s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import.hlotxt.test PASSED in 0.9s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import_async.hlotxt.test PASSED in 0.9s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:layouts_and_names.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:location.hlotxt.test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:module_attributes.hlo.test PASSED in 1.1s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:simple.hlo.test PASSED in 1.3s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:spmd_module_sharding.hlo.test PASSED in 1.0s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:types.hlotxt.test PASSED in 1.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:while.hlotxt.test PASSED in 1.0s //tensorflow/compiler/xla/translate/mhlo_to_hlo:type_to_shape_test PASSED in 1.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:add.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:case.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:dynamic.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export-with-layouts.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export.mlir.test PASSED in 1.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_and_check_layouts.mlir.test PASSED in 1.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_large_constants.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_replicas.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:frontend_attributes.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:fusion.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:if.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:input_output_aliasing.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:layouts_and_names.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_op_metadata.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:missing_main.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:module_attributes.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:multiple_return_tuple.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:opaque_elements_attr.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:rng_get_and_update_state.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:sharding.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:simple.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:unsupported_type.mlir.test PASSED in 1.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:while.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:hlo_text_to_lhlo_no_opt.hlotxt.test PASSED in 3.6s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:no_opt_ops.hlotxt.test PASSED in 1.6s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:non_identity_layouts.hlotxt.test PASSED in 1.4s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:ops.mlir.test PASSED in 5.3s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:passthrough.mlir.test PASSED in 0.8s //tensorflow/core:__tensorflow_core_lib_core_legacy_lib_core_all_tests PASSED in 18.6s //tensorflow/core:__tensorflow_core_lib_gtl_legacy_lib_gtl_tests PASSED in 0.8s //tensorflow/core:__tensorflow_core_lib_monitoring_cell_reader_test PASSED in 34.2s //tensorflow/core:__tensorflow_core_lib_monitoring_collection_registry_test PASSED in 0.5s //tensorflow/core:__tensorflow_core_lib_monitoring_counter_test PASSED in 0.4s //tensorflow/core:__tensorflow_core_lib_monitoring_gauge_test PASSED in 0.3s //tensorflow/core:__tensorflow_core_lib_monitoring_metric_def_test PASSED in 0.8s //tensorflow/core:__tensorflow_core_lib_monitoring_percentile_sampler_test PASSED in 1.1s //tensorflow/core:__tensorflow_core_lib_monitoring_sampler_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_test_utils_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_strings_legacy_low_level_library_tests PASSED in 0.7s //tensorflow/core:__tensorflow_core_lib_wav_wav_io_test PASSED in 0.3s //tensorflow/core:__tensorflow_core_util_mkl_util_test_srcs PASSED in 0.6s //tensorflow/core:__tensorflow_tsl_lib_core_legacy_lib_core_all_tests PASSED in 0.7s //tensorflow/core:lib_strings_ordered_code_test PASSED in 2.4s //tensorflow/core:lib_strings_proto_serialization_test PASSED in 0.2s //tensorflow/core/api_def:api_test PASSED in 3.6s //tensorflow/core/api_def:update_api_def_test PASSED in 0.8s //tensorflow/core/common_runtime:all_to_all_test_cpu PASSED in 0.8s //tensorflow/core/common_runtime:arg_ret_placement_test PASSED in 0.7s //tensorflow/core/common_runtime:buf_rendezvous_test PASSED in 1.7s //tensorflow/core/common_runtime:collective_executor_mgr_test PASSED in 1.0s //tensorflow/core/common_runtime:collective_param_resolver_local_test PASSED in 6.8s //tensorflow/core/common_runtime:collective_rma_local_test PASSED in 2.4s //tensorflow/core/common_runtime:composite_device_test PASSED in 1.7s //tensorflow/core/common_runtime:cost_measurement_registry_test PASSED in 2.4s //tensorflow/core/common_runtime:cost_util_test PASSED in 0.1s //tensorflow/core/common_runtime:device_mgr_test PASSED in 1.2s //tensorflow/core/common_runtime:device_propagation_test PASSED in 0.6s //tensorflow/core/common_runtime:device_resolver_local_test PASSED in 3.8s //tensorflow/core/common_runtime:device_set_test PASSED in 1.2s //tensorflow/core/common_runtime:direct_session_test_cpu PASSED in 3.5s //tensorflow/core/common_runtime:direct_session_with_debug_test PASSED in 5.1s //tensorflow/core/common_runtime:direct_session_with_tracking_alloc_test PASSED in 1.9s //tensorflow/core/common_runtime:dynamic_device_mgr_test PASSED in 1.6s //tensorflow/core/common_runtime:eval_const_tensor_test PASSED in 1.3s //tensorflow/core/common_runtime:executor_test PASSED in 2.8s //tensorflow/core/common_runtime:function_optimization_registration_test PASSED in 0.8s //tensorflow/core/common_runtime:function_optimization_registry_no_pass_test PASSED in 0.8s //tensorflow/core/common_runtime:function_optimization_registry_pass_failure_test PASSED in 2.1s //tensorflow/core/common_runtime:function_optimization_registry_test PASSED in 1.3s //tensorflow/core/common_runtime:function_threadpool_test PASSED in 1.5s //tensorflow/core/common_runtime:graph_constructor_test PASSED in 3.2s //tensorflow/core/common_runtime:graph_runner_test PASSED in 1.6s //tensorflow/core/common_runtime:hierarchical_tree_broadcaster_test_cpu PASSED in 4.4s //tensorflow/core/common_runtime:inline_function_utils_test PASSED in 0.9s //tensorflow/core/common_runtime:input_colocation_exemption_registry_test PASSED in 1.4s //tensorflow/core/common_runtime:int32_fulltype_test PASSED in 1.1s //tensorflow/core/common_runtime:isolate_placer_inspection_required_ops_pass_test PASSED in 1.0s //tensorflow/core/common_runtime:lower_case_op_test PASSED in 4.1s //tensorflow/core/common_runtime:lower_function_call_test PASSED in 2.6s //tensorflow/core/common_runtime:lower_functional_ops_test PASSED in 2.3s //tensorflow/core/common_runtime:lower_if_op_test PASSED in 3.9s //tensorflow/core/common_runtime:lower_while_op_test PASSED in 2.9s //tensorflow/core/common_runtime:mkl_cpu_allocator_test PASSED in 0.7s //tensorflow/core/common_runtime:mkl_threadpool_device_test PASSED in 0.5s //tensorflow/core/common_runtime:no_op_cost_measurement_test PASSED in 0.6s //tensorflow/core/common_runtime:null_request_cost_accessor_test PASSED in 0.2s //tensorflow/core/common_runtime:optimization_registry_test PASSED in 1.0s //tensorflow/core/common_runtime:optimize_cross_host_control_deps_test PASSED in 6.7s //tensorflow/core/common_runtime:optimize_function_graph_utils_test PASSED in 0.6s //tensorflow/core/common_runtime:partitioning_utils_test PASSED in 1.7s //tensorflow/core/common_runtime:pending_counts_test PASSED in 1.4s //tensorflow/core/common_runtime:permuter_test_cpu PASSED in 5.1s //tensorflow/core/common_runtime:placer_inspection_required_ops_utils_test PASSED in 0.9s //tensorflow/core/common_runtime:placer_test PASSED in 1.5s //tensorflow/core/common_runtime:process_function_library_runtime_test_cpu PASSED in 0.7s //tensorflow/core/common_runtime:process_util_test PASSED in 0.3s //tensorflow/core/common_runtime:quantize_training_test PASSED in 3.7s //tensorflow/core/common_runtime:rendezvous_util_test PASSED in 0.1s //tensorflow/core/common_runtime:replicate_per_replica_nodes_test PASSED in 0.9s //tensorflow/core/common_runtime:request_cost_accessor_registry_test PASSED in 2.5s //tensorflow/core/common_runtime:request_cost_test PASSED in 0.5s //tensorflow/core/common_runtime:ring_gatherer_test_cpu PASSED in 4.1s //tensorflow/core/common_runtime:ring_reducer_test_cpu PASSED in 6.8s //tensorflow/core/common_runtime:scoped_allocator_mgr_test PASSED in 4.0s //tensorflow/core/common_runtime:session_test PASSED in 1.2s //tensorflow/core/common_runtime:shape_refiner_test PASSED in 1.2s //tensorflow/core/common_runtime:single_threaded_executor_test PASSED in 0.9s //tensorflow/core/common_runtime:threadpool_device_test PASSED in 0.8s //tensorflow/core/common_runtime:type_inference_test PASSED in 3.1s //tensorflow/core/common_runtime/eager:attr_builder_test PASSED in 42.2s //tensorflow/core/common_runtime/eager:context_test PASSED in 20.7s //tensorflow/core/common_runtime/eager:custom_device_test PASSED in 21.8s //tensorflow/core/common_runtime/eager:eager_executor_test PASSED in 19.2s //tensorflow/core/common_runtime/eager:eager_op_rewrite_registry_test PASSED in 2.0s //tensorflow/core/common_runtime/eager:eager_operation_test PASSED in 20.9s //tensorflow/core/common_runtime/eager:execute_node_test PASSED in 17.8s //tensorflow/core/common_runtime/eager:execute_test PASSED in 44.5s //tensorflow/core/common_runtime/eager:kernel_and_device_test PASSED in 1.4s //tensorflow/core/common_runtime/eager:mkl_eager_op_rewrite_test PASSED in 19.0s //tensorflow/core/common_runtime/eager:placement_test PASSED in 17.9s //tensorflow/core/common_runtime/eager:placement_utils_test PASSED in 16.9s //tensorflow/core/common_runtime/eager:tensor_handle_data_test PASSED in 18.3s //tensorflow/core/common_runtime/eager:tensor_handle_test PASSED in 16.7s //tensorflow/core/common_runtime/gpu:gpu_device_on_non_gpu_machine_test PASSED in 0.7s //tensorflow/core/common_runtime/next_pluggable_device/c:plugin_c_api_test PASSED in 45.8s //tensorflow/core/config:flags_py_test PASSED in 6.4s //tensorflow/core/config:flags_test PASSED in 0.3s //tensorflow/core/data:compression_utils_test PASSED in 2.2s //tensorflow/core/data:dataset_utils_test PASSED in 1.1s //tensorflow/core/data:hash_utils_test PASSED in 2.2s //tensorflow/core/data:metric_utils_test PASSED in 6.1s //tensorflow/core/data:name_utils_test PASSED in 0.5s //tensorflow/core/data:rewrite_utils_test PASSED in 1.8s //tensorflow/core/data:serialization_utils_test PASSED in 0.5s //tensorflow/core/data:snapshot_utils_test PASSED in 2.4s //tensorflow/core/data:split_utils_test PASSED in 0.8s //tensorflow/core/data:standalone_save_restore_test PASSED in 3.5s //tensorflow/core/data:standalone_test PASSED in 3.5s //tensorflow/core/data:tfdataz_metrics_test PASSED in 2.5s //tensorflow/core/data:unbounded_thread_pool_test PASSED in 0.6s //tensorflow/core/data/service:auto_shard_rewriter_test PASSED in 1.7s //tensorflow/core/data/service:common_test PASSED in 0.7s //tensorflow/core/data/service:credentials_factory_test PASSED in 1.6s //tensorflow/core/data/service:cross_trainer_cache_test PASSED in 1.7s //tensorflow/core/data/service:data_service_test PASSED in 16.8s //tensorflow/core/data/service:data_transfer_test PASSED in 1.5s //tensorflow/core/data/service:dataset_store_test PASSED in 0.8s //tensorflow/core/data/service:dispatcher_client_test PASSED in 6.6s //tensorflow/core/data/service:dispatcher_state_test PASSED in 1.5s //tensorflow/core/data/service:grpc_dispatcher_impl_test PASSED in 4.3s //tensorflow/core/data/service:grpc_util_test PASSED in 2.1s //tensorflow/core/data/service:grpc_worker_impl_test PASSED in 6.2s //tensorflow/core/data/service:journal_test PASSED in 0.9s //tensorflow/core/data/service:logging_utils_test PASSED in 0.3s //tensorflow/core/data/service:task_runner_test PASSED in 6.5s //tensorflow/core/data/service:test_util_test PASSED in 3.6s //tensorflow/core/data/service:url_test PASSED in 0.1s //tensorflow/core/data/service:utils_test PASSED in 0.7s //tensorflow/core/data/service:validate_utils_test PASSED in 0.6s //tensorflow/core/data/service:worker_client_test PASSED in 5.8s //tensorflow/core/data/service:worker_impl_test PASSED in 6.3s //tensorflow/core/data/service/client:data_service_client_test PASSED in 7.1s //tensorflow/core/data/service/client:utils_test PASSED in 4.0s //tensorflow/core/data/service/client:validate_utils_test PASSED in 2.9s //tensorflow/core/data/service/snapshot:distributed_snapshot_test PASSED in 22.6s //tensorflow/core/data/service/snapshot:file_utils_test PASSED in 1.0s //tensorflow/core/data/service/snapshot:path_utils_test PASSED in 0.2s //tensorflow/core/data/service/snapshot:snapshot_manager_test PASSED in 3.9s //tensorflow/core/data/service/snapshot:snapshot_split_provider_test PASSED in 1.3s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_checkpoint_test PASSED in 5.0s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_test PASSED in 5.2s //tensorflow/core/data/service/snapshot:utils_test PASSED in 0.3s //tensorflow/core/debug:debug_graph_utils_test PASSED in 0.6s //tensorflow/core/distributed_runtime:call_options_test PASSED in 0.2s //tensorflow/core/distributed_runtime:cluster_function_library_runtime_test PASSED in 6.5s //tensorflow/core/distributed_runtime:collective_param_resolver_distributed_test PASSED in 2.6s //tensorflow/core/distributed_runtime:collective_rma_distributed_test PASSED in 0.7s //tensorflow/core/distributed_runtime:device_resolver_distributed_test PASSED in 1.1s //tensorflow/core/distributed_runtime:message_wrappers_test PASSED in 0.7s //tensorflow/core/distributed_runtime:partial_run_mgr_test PASSED in 0.5s //tensorflow/core/distributed_runtime:recent_request_ids_test PASSED in 0.4s //tensorflow/core/distributed_runtime:request_id_test PASSED in 0.1s //tensorflow/core/distributed_runtime:rpc_collective_executor_mgr_test PASSED in 0.9s //tensorflow/core/distributed_runtime:server_lib_test PASSED in 0.2s //tensorflow/core/distributed_runtime:session_mgr_test PASSED in 1.3s //tensorflow/core/distributed_runtime:tensor_coding_test PASSED in 0.6s //tensorflow/core/distributed_runtime/coordination:coordination_service_barrier_proxy_test PASSED in 2.8s //tensorflow/core/distributed_runtime/eager:eager_service_impl_test PASSED in 33.0s //tensorflow/core/distributed_runtime/eager:remote_mgr_test PASSED in 16.6s //tensorflow/core/distributed_runtime/integration_test:c_api_multi_client_test_cpu PASSED in 42.4s //tensorflow/core/distributed_runtime/integration_test:c_api_recoverable_jobs_test_cpu PASSED in 57.1s //tensorflow/core/distributed_runtime/integration_test:c_api_session_coordination_test_cpu PASSED in 39.8s //tensorflow/core/distributed_runtime/rpc:grpc_tensor_coding_test PASSED in 3.0s //tensorflow/core/distributed_runtime/rpc:grpc_worker_cache_test PASSED in 0.5s //tensorflow/core/distributed_runtime/rpc/eager:grpc_eager_client_test PASSED in 1.7s //tensorflow/core/example:example_parser_configuration_test PASSED in 1.6s //tensorflow/core/example:feature_util_test PASSED in 0.2s //tensorflow/core/framework:allocator_test PASSED in 4.5s //tensorflow/core/framework:attr_value_util_test PASSED in 1.0s //tensorflow/core/framework:batch_util_test PASSED in 1.0s //tensorflow/core/framework:bfloat16_test PASSED in 1.1s //tensorflow/core/framework:common_shape_fns_test PASSED in 1.8s //tensorflow/core/framework:dataset_test PASSED in 1.8s //tensorflow/core/framework:device_base_test PASSED in 0.9s //tensorflow/core/framework:disable_jit_test PASSED in 1.4s //tensorflow/core/framework:framework_op_gen_lib_test PASSED in 0.2s //tensorflow/core/framework:framework_op_segment_test PASSED in 1.9s //tensorflow/core/framework:framework_resource_var_test PASSED in 0.4s //tensorflow/core/framework:framework_run_handler_test PASSED in 4.0s //tensorflow/core/framework:framework_run_handler_util_test PASSED in 6.2s //tensorflow/core/framework:full_type_inference_util_test PASSED in 1.9s //tensorflow/core/framework:full_type_util_test PASSED in 1.8s //tensorflow/core/framework:function_test PASSED in 1.2s //tensorflow/core/framework:graph_def_util_test PASSED in 1.6s //tensorflow/core/framework:graph_to_functiondef_test PASSED in 1.8s //tensorflow/core/framework:kernel_def_builder_test PASSED in 0.9s //tensorflow/core/framework:kernel_def_util_test PASSED in 2.6s //tensorflow/core/framework:memory_types_test PASSED in 1.6s //tensorflow/core/framework:model_test PASSED in 1.0s //tensorflow/core/framework:node_def_builder_test PASSED in 1.4s //tensorflow/core/framework:node_def_util_test PASSED in 1.1s //tensorflow/core/framework:node_properties_test PASSED in 0.9s //tensorflow/core/framework:op_compatibility_test PASSED in 1.9s //tensorflow/core/framework:op_def_builder_test PASSED in 2.0s //tensorflow/core/framework:op_def_util_test PASSED in 1.0s //tensorflow/core/framework:op_kernel_test PASSED in 1.3s //tensorflow/core/framework:op_registration_test PASSED in 1.0s //tensorflow/core/framework:partial_tensor_shape_test PASSED in 0.9s //tensorflow/core/framework:rendezvous_test PASSED in 4.4s //tensorflow/core/framework:resource_handle_test PASSED in 0.7s //tensorflow/core/framework:resource_mgr_test PASSED in 2.7s //tensorflow/core/framework:resource_op_kernel_test PASSED in 1.7s //tensorflow/core/framework:shape_inference_test PASSED in 1.9s //tensorflow/core/framework:shape_inference_testutil_test PASSED in 1.4s //tensorflow/core/framework:tensor_shape_test PASSED in 8.0s //tensorflow/core/framework:tensor_slice_test PASSED in 1.1s //tensorflow/core/framework:tensor_test PASSED in 29.6s //tensorflow/core/framework:tensor_testutil_test PASSED in 1.1s //tensorflow/core/framework:tensor_util_test PASSED in 1.7s //tensorflow/core/framework:tracking_allocator_test PASSED in 0.8s //tensorflow/core/framework:types_test PASSED in 0.7s //tensorflow/core/framework:variant_op_registry_test PASSED in 17.9s //tensorflow/core/framework:variant_test PASSED in 0.9s //tensorflow/core/framework/registration:registration_test PASSED in 1.1s //tensorflow/core/function/capture:by_ref_capture_test PASSED in 11.1s //tensorflow/core/function/capture:capture_container_test PASSED in 6.0s //tensorflow/core/function/integration_test:side_inputs_manual_api_test PASSED in 14.3s //tensorflow/core/function/integration_test:side_inputs_test PASSED in 18.0s //tensorflow/core/function/polymorphism:function_cache_test PASSED in 6.7s //tensorflow/core/function/polymorphism:function_type_test PASSED in 6.6s //tensorflow/core/function/polymorphism:type_dispatch_test PASSED in 7.0s //tensorflow/core/function/runtime_client:runtime_client_cc_test PASSED in 60.4s //tensorflow/core/function/trace_type:default_types_test PASSED in 7.4s //tensorflow/core/function/trace_type:serialization_test PASSED in 6.7s //tensorflow/core/function/trace_type:trace_type_test PASSED in 10.8s //tensorflow/core/graph:algorithm_test PASSED in 1.3s //tensorflow/core/graph:collective_order_test PASSED in 1.0s //tensorflow/core/graph:control_flow_test PASSED in 1.6s //tensorflow/core/graph:costmodel_test PASSED in 1.1s //tensorflow/core/graph:edgeset_test PASSED in 1.7s //tensorflow/core/graph:graph_debug_info_builder_test PASSED in 0.7s //tensorflow/core/graph:graph_def_builder_test PASSED in 0.9s //tensorflow/core/graph:graph_partition_test PASSED in 2.1s //tensorflow/core/graph:graph_test PASSED in 1.8s //tensorflow/core/graph:node_builder_test PASSED in 1.8s //tensorflow/core/graph:optimizer_cse_test PASSED in 0.9s //tensorflow/core/graph:subgraph_test PASSED in 2.4s //tensorflow/core/graph:tensor_id_test PASSED in 1.6s //tensorflow/core/graph:validate_test PASSED in 1.6s //tensorflow/core/graph/regularization:simple_delete_test PASSED in 0.5s //tensorflow/core/graph/regularization:util_test PASSED in 0.3s //tensorflow/core/grappler:graph_topology_view_test PASSED in 0.2s //tensorflow/core/grappler:graph_view_test PASSED in 2.2s //tensorflow/core/grappler:grappler_item_builder_test PASSED in 3.2s //tensorflow/core/grappler:grappler_item_test PASSED in 2.3s //tensorflow/core/grappler:mutable_graph_view_test PASSED in 3.4s //tensorflow/core/grappler:utils_test PASSED in 3.8s //tensorflow/core/grappler/clusters:single_machine_test PASSED in 23.5s //tensorflow/core/grappler/clusters:virtual_cluster_test PASSED in 2.0s //tensorflow/core/grappler/costs:analytical_cost_estimator_test PASSED in 2.0s //tensorflow/core/grappler/costs:cost_estimator_test PASSED in 0.6s //tensorflow/core/grappler/costs:graph_memory_test PASSED in 4.2s //tensorflow/core/grappler/costs:graph_properties_test PASSED in 11.2s //tensorflow/core/grappler/costs:robust_stats_test PASSED in 0.3s //tensorflow/core/grappler/costs:utils_test PASSED in 3.4s //tensorflow/core/grappler/costs:virtual_placer_test PASSED in 0.7s //tensorflow/core/grappler/costs:virtual_scheduler_test PASSED in 2.8s //tensorflow/core/grappler/graph_analyzer:gen_node_test PASSED in 2.0s //tensorflow/core/grappler/graph_analyzer:graph_analyzer_test PASSED in 2.5s //tensorflow/core/grappler/graph_analyzer:hash_tools_test PASSED in 2.9s //tensorflow/core/grappler/graph_analyzer:sig_node_test PASSED in 3.1s //tensorflow/core/grappler/graph_analyzer:subgraph_test PASSED in 2.8s //tensorflow/core/grappler/inputs:utils_test PASSED in 0.3s //tensorflow/core/grappler/optimizers:arithmetic_optimizer_test_cpu PASSED in 5.3s //tensorflow/core/grappler/optimizers:auto_parallel_test_cpu PASSED in 2.7s //tensorflow/core/grappler/optimizers:common_subgraph_elimination_test_cpu PASSED in 3.8s //tensorflow/core/grappler/optimizers:custom_graph_optimizer_registry_test_cpu PASSED in 4.5s //tensorflow/core/grappler/optimizers:debug_stripper_test_cpu PASSED in 2.2s //tensorflow/core/grappler/optimizers:dependency_optimizer_test_cpu PASSED in 2.8s //tensorflow/core/grappler/optimizers:evaluation_utils_test PASSED in 1.0s //tensorflow/core/grappler/optimizers:function_api_info_test PASSED in 0.2s //tensorflow/core/grappler/optimizers:function_optimizer_test_cpu PASSED in 5.9s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_test_cpu PASSED in 2.6s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_factory_test PASSED in 0.5s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_test_cpu PASSED in 2.1s //tensorflow/core/grappler/optimizers:graph_optimizer_stage_test_cpu PASSED in 1.8s //tensorflow/core/grappler/optimizers:implementation_selector_test PASSED in 3.6s //tensorflow/core/grappler/optimizers:loop_optimizer_test_cpu PASSED in 2.3s //tensorflow/core/grappler/optimizers:memory_optimizer_test_cpu PASSED in 3.6s //tensorflow/core/grappler/optimizers:meta_optimizer_test_cpu PASSED in 8.9s //tensorflow/core/grappler/optimizers:mkl_remapper_test PASSED in 3.4s //tensorflow/core/grappler/optimizers:model_pruner_test_cpu PASSED in 2.3s //tensorflow/core/grappler/optimizers:pin_to_host_optimizer_test_cpu PASSED in 3.2s //tensorflow/core/grappler/optimizers:scoped_allocator_optimizer_test PASSED in 2.7s //tensorflow/core/grappler/optimizers:shape_optimizer_test_cpu PASSED in 2.8s //tensorflow/core/grappler/optimizers:static_schedule_test_cpu PASSED in 2.4s //tensorflow/core/grappler/optimizers:tfg_optimizer_hook_test PASSED in 1.1s //tensorflow/core/grappler/optimizers/data:auto_shard_test PASSED in 1.2s //tensorflow/core/grappler/optimizers/data:autotune_buffer_sizes_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:batch_parallelization_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:disable_intra_op_parallelism_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:disable_prefetch_legacy_autotune_test PASSED in 1.5s //tensorflow/core/grappler/optimizers/data:enable_gradient_descent_test PASSED in 0.9s //tensorflow/core/grappler/optimizers/data:filter_fusion_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:filter_parallelization_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:function_utils_test PASSED in 0.9s //tensorflow/core/grappler/optimizers/data:fusion_utils_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:graph_utils_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:inject_prefetch_test PASSED in 1.0s //tensorflow/core/grappler/optimizers/data:make_deterministic_test PASSED in 0.9s //tensorflow/core/grappler/optimizers/data:make_sloppy_test PASSED in 1.3s //tensorflow/core/grappler/optimizers/data:map_and_batch_fusion_test PASSED in 0.9s //tensorflow/core/grappler/optimizers/data:map_and_filter_fusion_test PASSED in 1.2s //tensorflow/core/grappler/optimizers/data:map_fusion_test PASSED in 0.9s //tensorflow/core/grappler/optimizers/data:map_parallelization_test PASSED in 1.9s //tensorflow/core/grappler/optimizers/data:noop_elimination_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:parallel_batch_test PASSED in 1.4s //tensorflow/core/grappler/optimizers/data:replicate_on_split_test PASSED in 1.2s //tensorflow/core/grappler/optimizers/data:shuffle_and_repeat_fusion_test PASSED in 0.9s //tensorflow/core/grappler/optimizers/data:slack_test PASSED in 1.2s //tensorflow/core/grappler/optimizers/data:split_utils_test PASSED in 2.3s //tensorflow/core/grappler/optimizers/data:use_private_thread_pool_test PASSED in 1.4s //tensorflow/core/grappler/optimizers/inference:batch_op_rewriter_test PASSED in 0.4s //tensorflow/core/grappler/utils:canonicalizer_test PASSED in 2.5s //tensorflow/core/grappler/utils:colocation_test PASSED in 1.0s //tensorflow/core/grappler/utils:frame_test PASSED in 0.2s //tensorflow/core/grappler/utils:functions_test PASSED in 4.8s //tensorflow/core/grappler/utils:graph_view_internal_test PASSED in 1.6s //tensorflow/core/grappler/utils:graph_view_test PASSED in 3.6s //tensorflow/core/grappler/utils:grappler_test_test PASSED in 8.2s //tensorflow/core/grappler/utils:pattern_utils_test PASSED in 1.2s //tensorflow/core/grappler/utils:scc_test PASSED in 3.1s //tensorflow/core/grappler/utils:symbolic_shapes_test PASSED in 0.3s //tensorflow/core/grappler/utils:topological_sort_test PASSED in 0.7s //tensorflow/core/grappler/utils:tpu_test PASSED in 0.1s //tensorflow/core/grappler/utils:transitive_fanin_test PASSED in 0.8s //tensorflow/core/grappler/utils:traversal_test PASSED in 1.3s //tensorflow/core/grappler/verifiers:structure_verifier_test PASSED in 1.7s //tensorflow/core/ir:interfaces_test PASSED in 0.8s //tensorflow/core/ir:ops_test PASSED in 0.3s //tensorflow/core/ir:shape_inference_utils_test PASSED in 0.6s //tensorflow/core/ir:tf_op_registry_test PASSED in 0.3s //tensorflow/core/ir:tf_op_wrapper_test PASSED in 0.8s //tensorflow/core/ir:utility_test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:arg_as_control_ret.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:backedge_segment.pbtxt.test PASSED in 2.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:empty.pbtxt.test PASSED in 1.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:error_during_backedge.pbtxt.test PASSED in 1.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_case_with_attr_inference.pbtxt.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_if_with_attr_inference.pbtxt.test PASSED in 1.3s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_iterator_get_next_attr_inference.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_underscore_output_shapes.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_while_with_attr_inference.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infeed_dequeue.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_arg_handle_type.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_with_output_shapes.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_arg_name.pbtxt.test PASSED in 1.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_backedge_input_size.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_duplicated_node_name.pbtxt.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_index.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_name.pbtxt.test PASSED in 1.3s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_attr_key.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_key.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_name.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_op_type.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_func_with_empty_name.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_function_import.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_control_result.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_input.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_name.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_result.pbtxt.test PASSED in 1.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_attr_name.pbtxt.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_named_edge_index.pbtxt.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_handle_data.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_input.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result.pbtxt.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result_value.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result_value.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_input.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_two_inputs.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_named_edge_index.pbtxt.test PASSED in 1.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_op_name.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_type_list.pbtxt.test PASSED in 1.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:legacy_call.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_shape.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_zero_constant.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:three_nodes_with_attrs.pbtxt.test PASSED in 1.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:version.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:empty.mlir.test PASSED in 1.5s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:fulltype.mlir.test PASSED in 2.7s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:func_with_no_args_or_results.mlir.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:negative_zero_constant.mlir.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:nested_legacy_call.mlir.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:three_nodes_with_attrs.mlir.test PASSED in 1.6s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:version.mlir.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/saved_model:saved_model_roundtrip_test PASSED in 0.5s //tensorflow/core/ir/tests:attributes.mlir.test PASSED in 1.0s //tensorflow/core/ir/tests:canonicalize.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:compatible_types.mlir.test PASSED in 1.0s //tensorflow/core/ir/tests:concrete-ops.mlir.test PASSED in 0.7s //tensorflow/core/ir/tests:generic_concrete_ops.mlir.test PASSED in 1.0s //tensorflow/core/ir/tests:invalid-concrete-ops.mlir.test PASSED in 0.9s //tensorflow/core/ir/tests:invalid-preserved-attrs.mlir.test PASSED in 0.4s //tensorflow/core/ir/tests:invalid.mlir.test PASSED in 1.1s //tensorflow/core/ir/tests:invalid_types.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:ops.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:region-invalid-ops.mlir.test PASSED in 1.3s //tensorflow/core/ir/tests:region-ops-graph.mlir.test PASSED in 1.6s //tensorflow/core/ir/tests:region-ops.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:types.mlir.test PASSED in 0.9s //tensorflow/core/ir/types:dialect_test PASSED in 0.7s //tensorflow/core/kernels:as_string_op_test PASSED in 1.5s //tensorflow/core/kernels:basic_ops_benchmark_test PASSED in 1.0s //tensorflow/core/kernels:batch_kernels_env_test PASSED in 1.0s //tensorflow/core/kernels:batch_kernels_test PASSED in 0.9s //tensorflow/core/kernels:bias_op_test PASSED in 1.2s //tensorflow/core/kernels:bincount_op_test_cpu PASSED in 1.2s //tensorflow/core/kernels:broadcast_to_op_test_cpu PASSED in 1.9s //tensorflow/core/kernels:cast_op_test_cpu PASSED in 2.2s //tensorflow/core/kernels:checkpoint_callback_manager_test PASSED in 0.8s //tensorflow/core/kernels:clustering_ops_test PASSED in 1.1s //tensorflow/core/kernels:composite_tensor_variant_test PASSED in 0.7s //tensorflow/core/kernels:concat_op_test PASSED in 1.0s //tensorflow/core/kernels:constant_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:control_flow_ops_test PASSED in 6.9s //tensorflow/core/kernels:conv_grad_filter_ops_benchmark_test_cpu PASSED in 0.9s //tensorflow/core/kernels:conv_grad_input_ops_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels:conv_ops_benchmark_test_cpu PASSED in 1.7s //tensorflow/core/kernels:conv_ops_test_cpu PASSED in 12.6s //tensorflow/core/kernels:count_ops_test PASSED in 1.6s //tensorflow/core/kernels:cross_op_test PASSED in 3.1s //tensorflow/core/kernels:cwise_ops_test_cpu PASSED in 1.0s //tensorflow/core/kernels:debug_ops_test PASSED in 1.1s //tensorflow/core/kernels:decode_wav_op_test PASSED in 3.1s //tensorflow/core/kernels:deep_conv2d_test PASSED in 1.2s //tensorflow/core/kernels:dequantize_op_test PASSED in 0.8s //tensorflow/core/kernels:diag_op_test_cpu PASSED in 1.0s //tensorflow/core/kernels:dynamic_partition_op_test_cpu PASSED in 1.5s //tensorflow/core/kernels:dynamic_stitch_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:eigen_activations_test PASSED in 0.6s //tensorflow/core/kernels:eigen_attention_test PASSED in 0.6s //tensorflow/core/kernels:eigen_backward_cuboid_convolutions_test PASSED in 1.3s //tensorflow/core/kernels:eigen_backward_spatial_convolutions_test PASSED in 0.5s //tensorflow/core/kernels:eigen_benchmark_cpu_test PASSED in 0.5s //tensorflow/core/kernels:eigen_mkldnn_contraction_kernel_test PASSED in 0.7s //tensorflow/core/kernels:eigen_pooling_test PASSED in 0.6s //tensorflow/core/kernels:encode_wav_op_test PASSED in 2.5s //tensorflow/core/kernels:fingerprint_op_test PASSED in 1.5s //tensorflow/core/kernels:fused_batch_norm_ex_op_test_cpu PASSED in 1.4s //tensorflow/core/kernels:fused_batch_norm_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:gather_nd_op_test_cpu PASSED in 1.2s //tensorflow/core/kernels:gather_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:guarantee_const_op_test PASSED in 1.0s //tensorflow/core/kernels:identity_n_op_test PASSED in 0.6s //tensorflow/core/kernels:identity_op_test PASSED in 0.5s //tensorflow/core/kernels:immutable_constant_op_test PASSED in 1.4s //tensorflow/core/kernels:in_topk_op_test PASSED in 0.8s //tensorflow/core/kernels:isotonic_regression_op_test PASSED in 1.3s //tensorflow/core/kernels:logging_ops_test PASSED in 2.9s //tensorflow/core/kernels:lookup_ops_test PASSED in 0.7s //tensorflow/core/kernels:loss_test PASSED in 0.1s //tensorflow/core/kernels:lrn_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:matmul_op_test_cpu PASSED in 6.8s //tensorflow/core/kernels:merge_v2_checkpoints_op_test PASSED in 1.2s //tensorflow/core/kernels:mfcc_dct_test PASSED in 0.5s //tensorflow/core/kernels:mfcc_mel_filterbank_test PASSED in 0.3s //tensorflow/core/kernels:mfcc_op_test_cpu PASSED in 2.7s //tensorflow/core/kernels:mfcc_test PASSED in 0.2s //tensorflow/core/kernels:multinomial_op_test_cpu PASSED in 1.2s //tensorflow/core/kernels:nn_ops_test_cpu PASSED in 0.7s //tensorflow/core/kernels:one_hot_op_test PASSED in 1.0s //tensorflow/core/kernels:ops_testutil_test PASSED in 0.8s //tensorflow/core/kernels:ops_util_test PASSED in 0.5s //tensorflow/core/kernels:parameterized_truncated_normal_op_test_cpu PASSED in 1.0s //tensorflow/core/kernels:parse_tensor_test PASSED in 1.5s //tensorflow/core/kernels:quantization_utils_test PASSED in 0.8s //tensorflow/core/kernels:quantize_and_dequantize_op_test_cpu PASSED in 1.8s //tensorflow/core/kernels:quantize_down_and_shrink_range_op_test PASSED in 1.3s //tensorflow/core/kernels:quantize_op_test PASSED in 1.5s //tensorflow/core/kernels:quantized_activation_ops_test PASSED in 1.2s //tensorflow/core/kernels:quantized_add_op_test PASSED in 1.8s //tensorflow/core/kernels:quantized_batch_norm_op_test PASSED in 1.3s //tensorflow/core/kernels:quantized_bias_add_op_test PASSED in 0.7s //tensorflow/core/kernels:quantized_concat_op_test PASSED in 0.7s //tensorflow/core/kernels:quantized_conv_ops_test PASSED in 2.1s //tensorflow/core/kernels:quantized_instance_norm_test PASSED in 1.7s //tensorflow/core/kernels:quantized_matmul_op_test PASSED in 0.9s //tensorflow/core/kernels:quantized_mul_op_test PASSED in 2.4s //tensorflow/core/kernels:quantized_pooling_ops_test PASSED in 1.2s //tensorflow/core/kernels:quantized_reshape_op_test PASSED in 1.0s //tensorflow/core/kernels:quantized_resize_bilinear_op_test PASSED in 2.5s //tensorflow/core/kernels:ragged_fill_empty_rows_op_test PASSED in 1.8s //tensorflow/core/kernels:ragged_gather_op_test PASSED in 1.1s //tensorflow/core/kernels:ragged_range_op_test PASSED in 1.7s //tensorflow/core/kernels:ragged_tensor_from_variant_op_test PASSED in 0.9s //tensorflow/core/kernels:ragged_tensor_to_sparse_kernel_test PASSED in 1.0s //tensorflow/core/kernels:ragged_tensor_to_tensor_op_test PASSED in 1.4s //tensorflow/core/kernels:ragged_tensor_to_variant_op_test PASSED in 1.8s //tensorflow/core/kernels:random_binomial_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:random_index_shuffle_test PASSED in 0.2s //tensorflow/core/kernels:random_op_test_cpu PASSED in 1.1s //tensorflow/core/kernels:random_poisson_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:range_sampler_test PASSED in 0.4s //tensorflow/core/kernels:reduction_ops_test_cpu PASSED in 1.2s //tensorflow/core/kernels:regex_replace_op_test PASSED in 0.6s //tensorflow/core/kernels:requantization_range_op_test PASSED in 1.0s //tensorflow/core/kernels:requantize_op_test PASSED in 0.8s //tensorflow/core/kernels:resource_ops_test PASSED in 1.2s //tensorflow/core/kernels:restore_op_test PASSED in 1.2s //tensorflow/core/kernels:restore_v2_op_test PASSED in 1.3s //tensorflow/core/kernels:reverse_op_test PASSED in 1.8s //tensorflow/core/kernels:roll_op_test PASSED in 1.5s //tensorflow/core/kernels:save_op_test PASSED in 1.1s //tensorflow/core/kernels:save_v2_op_test PASSED in 1.2s //tensorflow/core/kernels:scan_ops_test_cpu PASSED in 0.6s //tensorflow/core/kernels:scatter_nd_op_test_cpu PASSED in 1.1s //tensorflow/core/kernels:scatter_op_test PASSED in 1.0s //tensorflow/core/kernels:scoped_allocator_ops_test_cpu PASSED in 7.4s //tensorflow/core/kernels:sdca_ops_test PASSED in 2.0s //tensorflow/core/kernels:segment_reduction_ops_test PASSED in 1.3s //tensorflow/core/kernels:sendrecv_ops_test PASSED in 0.5s //tensorflow/core/kernels:sequence_ops_test PASSED in 1.4s //tensorflow/core/kernels:shape_ops_test PASSED in 1.4s //tensorflow/core/kernels:slice_op_test PASSED in 1.1s //tensorflow/core/kernels:spacetobatch_benchmark_test_cpu PASSED in 1.4s //tensorflow/core/kernels:sparse_add_op_test PASSED in 1.5s //tensorflow/core/kernels:sparse_dense_binary_op_shared_test PASSED in 0.8s //tensorflow/core/kernels:sparse_fill_empty_rows_op_test_cpu PASSED in 1.4s //tensorflow/core/kernels:sparse_matmul_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:sparse_reduce_sum_op_test PASSED in 0.7s //tensorflow/core/kernels:sparse_tensor_dense_matmul_op_test_cpu PASSED in 1.0s //tensorflow/core/kernels:sparse_to_dense_op_test_cpu PASSED in 1.2s //tensorflow/core/kernels:sparse_utils_test PASSED in 1.1s //tensorflow/core/kernels:sparse_xent_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:spectrogram_op_test_cpu PASSED in 5.9s //tensorflow/core/kernels:spectrogram_test PASSED in 0.1s //tensorflow/core/kernels:split_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:split_v_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:strided_slice_op_test PASSED in 0.7s //tensorflow/core/kernels:string_format_op_test PASSED in 0.6s //tensorflow/core/kernels:string_ngrams_op_test PASSED in 2.0s //tensorflow/core/kernels:string_split_op_test PASSED in 1.0s //tensorflow/core/kernels:substr_op_test PASSED in 0.9s //tensorflow/core/kernels:summary_audio_op_test PASSED in 2.3s //tensorflow/core/kernels:summary_image_op_test PASSED in 1.2s //tensorflow/core/kernels:summary_op_test PASSED in 1.0s //tensorflow/core/kernels:summary_tensor_op_test PASSED in 1.1s //tensorflow/core/kernels:tensor_cord_test PASSED in 0.8s //tensorflow/core/kernels:tensor_flag_utils_test PASSED in 0.6s //tensorflow/core/kernels:tensor_map_test PASSED in 0.2s //tensorflow/core/kernels:training_ops_test PASSED in 1.3s //tensorflow/core/kernels:transpose_util_test PASSED in 1.1s //tensorflow/core/kernels:unary_ops_composition_test_cpu PASSED in 3.5s //tensorflow/core/kernels:unique_op_test PASSED in 0.5s //tensorflow/core/kernels:variable_ops_test PASSED in 2.4s //tensorflow/core/kernels:while_op_test PASSED in 2.0s //tensorflow/core/kernels:xent_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels/batching_util:basic_batch_scheduler_test PASSED in 0.6s //tensorflow/core/kernels/batching_util:batch_input_task_test PASSED in 1.0s //tensorflow/core/kernels/batching_util:batch_resource_base_test PASSED in 0.8s //tensorflow/core/kernels/batching_util:batch_scheduler_test PASSED in 0.6s //tensorflow/core/kernels/batching_util:bounded_executor_test PASSED in 20.8s //tensorflow/core/kernels/batching_util:input_split_metadata_test PASSED in 0.7s //tensorflow/core/kernels/batching_util:periodic_function_test PASSED in 3.5s //tensorflow/core/kernels/batching_util:serial_device_batch_scheduler_test PASSED in 4.0s //tensorflow/core/kernels/batching_util:shared_batch_scheduler_test PASSED in 4.5s //tensorflow/core/kernels/batching_util:threadsafe_status_test PASSED in 0.4s //tensorflow/core/kernels/data:batch_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data:cache_dataset_ops_test PASSED in 1.7s //tensorflow/core/kernels/data:concatenate_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:filter_dataset_op_test PASSED in 2.5s //tensorflow/core/kernels/data:finalize_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:fixed_length_record_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:flat_map_dataset_op_test PASSED in 1.9s //tensorflow/core/kernels/data:get_options_op_test PASSED in 3.3s //tensorflow/core/kernels/data:interleave_dataset_op_test PASSED in 4.2s //tensorflow/core/kernels/data:iterator_ops_test PASSED in 2.0s //tensorflow/core/kernels/data:map_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:map_defun_op_test PASSED in 0.9s //tensorflow/core/kernels/data:optimize_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:options_dataset_op_test PASSED in 1.8s //tensorflow/core/kernels/data:padded_batch_dataset_op_test PASSED in 1.7s //tensorflow/core/kernels/data:parallel_batch_dataset_op_test PASSED in 3.3s //tensorflow/core/kernels/data:parallel_filter_dataset_op_test PASSED in 2.6s //tensorflow/core/kernels/data:parallel_interleave_dataset_op_test PASSED in 7.0s //tensorflow/core/kernels/data:parallel_map_dataset_op_test PASSED in 2.5s //tensorflow/core/kernels/data:prefetch_autotuner_test PASSED in 0.2s //tensorflow/core/kernels/data:prefetch_dataset_op_test PASSED in 1.7s //tensorflow/core/kernels/data:range_dataset_op_test PASSED in 2.0s //tensorflow/core/kernels/data:reduce_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:repeat_dataset_op_test PASSED in 3.6s //tensorflow/core/kernels/data:rewrite_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data:shard_dataset_op_test PASSED in 3.8s //tensorflow/core/kernels/data:shuffle_dataset_op_test PASSED in 1.7s //tensorflow/core/kernels/data:skip_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:sparse_tensor_slice_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:take_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:tensor_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:tensor_slice_dataset_op_test PASSED in 1.9s //tensorflow/core/kernels/data:text_line_dataset_op_test PASSED in 2.3s //tensorflow/core/kernels/data:tf_record_dataset_op_test PASSED in 1.6s //tensorflow/core/kernels/data:window_dataset_op_test PASSED in 1.4s //tensorflow/core/kernels/data:zip_dataset_op_test PASSED in 2.0s //tensorflow/core/kernels/data/experimental:assert_next_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data/experimental:assert_prev_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data/experimental:auto_shard_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data/experimental:directed_interleave_dataset_op_test PASSED in 3.9s //tensorflow/core/kernels/data/experimental:list_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data/experimental:map_and_batch_dataset_op_test PASSED in 2.9s //tensorflow/core/kernels/data/experimental:parallel_interleave_dataset_op_test PASSED in 3.1s //tensorflow/core/kernels/data/experimental:random_dataset_op_test PASSED in 2.5s //tensorflow/core/kernels/data/experimental:sampling_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data/experimental:save_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data/experimental:unique_dataset_op_test PASSED in 1.4s //tensorflow/core/kernels/image:adjust_contrast_op_benchmark_test_cpu PASSED in 0.7s //tensorflow/core/kernels/image:adjust_contrast_op_test PASSED in 0.6s //tensorflow/core/kernels/image:colorspace_op_test PASSED in 1.1s //tensorflow/core/kernels/image:crop_and_resize_op_benchmark_test_cpu PASSED in 0.8s //tensorflow/core/kernels/image:crop_and_resize_op_test PASSED in 1.2s //tensorflow/core/kernels/image:encode_jpeg_op_test PASSED in 0.7s //tensorflow/core/kernels/image:mirror_pad_op_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels/image:mirror_pad_op_test PASSED in 2.5s //tensorflow/core/kernels/image:non_max_suppression_op_benchmark_test PASSED in 1.2s //tensorflow/core/kernels/image:non_max_suppression_op_test PASSED in 1.5s //tensorflow/core/kernels/image:resize_area_op_test PASSED in 1.5s //tensorflow/core/kernels/image:resize_benchmark_test_cpu PASSED in 0.9s //tensorflow/core/kernels/image:resize_bicubic_op_test PASSED in 4.3s //tensorflow/core/kernels/image:resize_ops_test_cpu PASSED in 2.3s //tensorflow/core/kernels/image:sampling_kernels_test PASSED in 1.2s //tensorflow/core/kernels/image:scale_and_translate_op_test PASSED in 2.2s //tensorflow/core/kernels/linalg:banded_triangular_solve_op_test_cpu PASSED in 1.7s //tensorflow/core/kernels/linalg:matrix_triangular_solve_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels/mkl:mkl_conv_ops_test PASSED in 0.6s //tensorflow/core/kernels/mkl:mkl_dequantize_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_fused_batch_norm_op_test PASSED in 1.0s //tensorflow/core/kernels/mkl:mkl_fused_ops_test PASSED in 3.4s //tensorflow/core/kernels/mkl:mkl_matmul_op_benchmark PASSED in 0.3s //tensorflow/core/kernels/mkl:mkl_qmatmul_op_test PASSED in 0.5s //tensorflow/core/kernels/mkl:mkl_quantize_op_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_quantized_concat_op_test PASSED in 0.6s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_perchannel_test PASSED in 0.6s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_quantized_pooling_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_relu_op_test PASSED in 0.3s //tensorflow/core/kernels/mkl:mkl_requantize_ops_test PASSED in 0.3s //tensorflow/core/kernels/mkl:mkl_swish_op_test PASSED in 0.2s //tensorflow/core/kernels/mkl:onednn_nn_ops_benchmark PASSED in 0.4s //tensorflow/core/kernels/sparse:kernels_test PASSED in 0.8s //tensorflow/core/kernels/uniform_quant_ops:math_utils_test PASSED in 0.1s //tensorflow/core/kernels/uniform_quant_ops:tensor_utils_test PASSED in 0.2s //tensorflow/core/kernels/uniform_quant_ops:uniform_dequantize_op_test PASSED in 0.8s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantize_op_test PASSED in 0.6s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_add_op_test PASSED in 1.2s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_clip_by_value_op_test PASSED in 0.6s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_convolution_ops_test PASSED in 1.3s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_dot_ops_test PASSED in 2.4s //tensorflow/core/kernels/uniform_quant_ops:uniform_requantize_op_test PASSED in 1.5s //tensorflow/core/lib/db:sqlite_test PASSED in 0.3s //tensorflow/core/lib/gif:lib_gif_io_test PASSED in 3.6s //tensorflow/core/lib/jpeg:lib_jpeg_jpeg_mem_unittest PASSED in 0.8s //tensorflow/core/ops:cudnn_rnn_ops_test_cc PASSED in 0.6s //tensorflow/core/ops:ops_array_grad_test PASSED in 3.5s //tensorflow/core/ops:ops_math_grad_test PASSED in 7.6s //tensorflow/core/ops:ops_tests PASSED in 1.4s //tensorflow/core/ops/compat:backwards_compatibility_test PASSED in 1.3s //tensorflow/core/platform:__tensorflow_tsl_platform_profile_utils_cpu_utils_test PASSED in 0.1s //tensorflow/core/platform:enable_tf2_utils_test PASSED in 0.2s //tensorflow/core/platform:env_test PASSED in 4.3s //tensorflow/core/platform:fake_python_env_test PASSED in 0.1s //tensorflow/core/platform:file_system_test PASSED in 3.0s //tensorflow/core/platform:platform_strings_test PASSED in 0.2s //tensorflow/core/platform:ram_file_system_test PASSED in 35.1s //tensorflow/core/platform:resource_loader_test PASSED in 0.6s //tensorflow/core/platform:vmodule_benchmark_test PASSED in 0.2s //tensorflow/core/platform:vmodule_test PASSED in 0.4s //tensorflow/core/profiler/backends/cpu:host_tracer_test PASSED in 1.0s //tensorflow/core/profiler/convert:hlo_proto_to_graph_view_test PASSED in 0.2s //tensorflow/core/profiler/convert:hlo_proto_to_memory_visualization_utils_test PASSED in 0.2s //tensorflow/core/profiler/convert:op_stats_to_pod_stats_test PASSED in 0.6s //tensorflow/core/profiler/convert:op_stats_to_pod_viewer_test PASSED in 0.5s //tensorflow/core/profiler/convert:op_stats_to_tf_stats_test PASSED in 0.1s //tensorflow/core/profiler/convert:repository_test PASSED in 0.6s //tensorflow/core/profiler/convert:xplane_to_kernel_stats_db_test PASSED in 0.3s //tensorflow/core/profiler/convert:xplane_to_memory_profile_test PASSED in 0.5s //tensorflow/core/profiler/convert:xplane_to_op_metrics_db_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_op_stats_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_step_events_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_tf_functions_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_tool_names_test PASSED in 0.4s //tensorflow/core/profiler/convert/trace_viewer:trace_viewer_visibility_test PASSED in 0.4s //tensorflow/core/profiler/internal:tfprof_show_test PASSED in 1.5s //tensorflow/core/profiler/internal:tfprof_stats_test PASSED in 1.7s //tensorflow/core/profiler/internal:tfprof_tensor_test PASSED in 0.6s //tensorflow/core/profiler/internal:tfprof_timeline_test PASSED in 0.8s //tensorflow/core/profiler/internal/advisor:tfprof_advisor_test PASSED in 1.1s //tensorflow/core/profiler/lib:profiler_disabled_test PASSED in 1.0s //tensorflow/core/profiler/utils:derived_timeline_test PASSED in 0.3s //tensorflow/core/profiler/utils:kernel_stats_utils_test PASSED in 0.5s //tensorflow/core/profiler/utils:op_metrics_db_utils_test PASSED in 0.3s //tensorflow/core/profiler/utils:step_intersection_test PASSED in 0.3s //tensorflow/core/summary:schema_test PASSED in 0.1s //tensorflow/core/summary:summary_db_writer_test PASSED in 0.2s //tensorflow/core/summary:summary_file_writer_test PASSED in 0.5s //tensorflow/core/tfrt/common:pjrt_cpu_client_registration_test PASSED in 10.4s //tensorflow/core/tfrt/common:pjrt_state_test PASSED in 11.1s //tensorflow/core/tfrt/common:pjrt_util_test PASSED in 10.2s //tensorflow/core/tfrt/fallback:cost_recorder_test PASSED in 0.2s //tensorflow/core/tfrt/fallback:fallback_state_test PASSED in 0.8s //tensorflow/core/transforms:eval_utils_test PASSED in 2.4s //tensorflow/core/transforms:graph_transform_wrapper_test PASSED in 0.6s //tensorflow/core/util:bcast_test PASSED in 0.8s //tensorflow/core/util:command_line_flags_test PASSED in 1.0s //tensorflow/core/util:debug_data_dumper_test PASSED in 1.3s //tensorflow/core/util:debug_events_writer_test PASSED in 0.4s //tensorflow/core/util:dump_graph_test PASSED in 0.9s //tensorflow/core/util:equal_graph_def_test PASSED in 1.3s //tensorflow/core/util:events_writer_test PASSED in 3.1s //tensorflow/core/util:example_proto_fast_parsing_test PASSED in 1.1s //tensorflow/core/util:example_proto_helper_test PASSED in 1.2s //tensorflow/core/util:exec_on_stall_test PASSED in 2.1s //tensorflow/core/util:fake_clock_env_test PASSED in 2.0s //tensorflow/core/util:incremental_barrier_test PASSED in 0.1s //tensorflow/core/util:matmul_bcast_test PASSED in 2.1s //tensorflow/core/util:memmapped_file_system_test PASSED in 2.1s //tensorflow/core/util:overflow_test PASSED in 0.4s //tensorflow/core/util:presized_cuckoo_map_test PASSED in 2.1s //tensorflow/core/util:ragged_to_dense_util_test PASSED in 1.3s //tensorflow/core/util:reffed_status_callback_test PASSED in 2.2s //tensorflow/core/util:reporter_test PASSED in 1.7s //tensorflow/core/util:saved_tensor_slice_util_test PASSED in 1.4s //tensorflow/core/util:semver_test PASSED in 1.5s //tensorflow/core/util:stat_summarizer_test PASSED in 1.8s //tensorflow/core/util:strided_slice_op_test PASSED in 0.9s //tensorflow/core/util:tensor_format_test PASSED in 1.0s //tensorflow/core/util:tensor_slice_reader_test PASSED in 1.1s //tensorflow/core/util:tensor_slice_set_test PASSED in 1.5s //tensorflow/core/util:tensor_slice_util_test PASSED in 1.7s //tensorflow/core/util:tensor_slice_writer_test PASSED in 1.5s //tensorflow/core/util:work_sharder_test PASSED in 0.9s //tensorflow/core/util/ctc:ctc_beam_search_test PASSED in 0.1s //tensorflow/core/util/proto:descriptor_pool_registry_test PASSED in 0.8s //tensorflow/core/util/proto:proto_utils_test PASSED in 1.0s //tensorflow/core/util/quantization:uniform_quant_ops_params_test PASSED in 0.9s //tensorflow/core/util/sparse:sparse_tensor_test PASSED in 0.6s //tensorflow/core/util/tensor_bundle:tensor_bundle_test PASSED in 36.5s //tensorflow/dtensor/mlir:dtensor_location_test PASSED in 0.5s //tensorflow/dtensor/mlir:group_assignment_test PASSED in 0.4s //tensorflow/dtensor/mlir/tests:annotate_global_shape.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:cluster_function_conversion.mlir.test PASSED in 1.6s //tensorflow/dtensor/mlir/tests:constant_folding.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:designate_resource_handle_mesh.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:device_mesh_cluster_coarsening.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_all_gather.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_all_scatter.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_combine_optimization.mlir.test PASSED in 1.4s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_lowering.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_scatter_optimization.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_sum_optimization.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_alltoall_lowering.mlir.test PASSED in 1.2s //tensorflow/dtensor/mlir/tests:dtensor_layout_must_execute.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_layout_to_xla_sharding_op.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_mixed_precision_reduce.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:dtensor_reduce_scatter_lowering.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_remove_dtensorlayout.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:dtensor_replace_auxiliary_layout_op.mlir.test PASSED in 1.7s //tensorflow/dtensor/mlir/tests:dtensor_replace_relayout_with_identity.mlir.test PASSED in 1.2s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding_default.mlir.test PASSED in 0.4s //tensorflow/dtensor/mlir/tests:dtensor_xla_spmd_integration.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:elide_identity_before_copy_to_mesh.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:function_renaming.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:handle_cross_cluster_dependencies.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:handle_sparsetensors.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:layout_propagation_v2.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:lower_send_recv.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:merge_clusters.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:mesh_propagation.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:multi_device_expansion.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:op_to_device_cluster.mlir.test PASSED in 1.8s //tensorflow/dtensor/mlir/tests:propagate_default_layout.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:propagate_device_id_to_function.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:restore_and_assign.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:restore_shape_inference.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:set_default_sharding.mlir.test PASSED in 1.2s //tensorflow/dtensor/mlir/tests:sparse_expansion.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:spmd_batchparallel.mlir.test PASSED in 1.7s //tensorflow/dtensor/mlir/tests:spmd_concat.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_conv.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:spmd_einsum.mlir.test PASSED in 1.8s //tensorflow/dtensor/mlir/tests:spmd_expansion.mlir.test PASSED in 1.5s //tensorflow/dtensor/mlir/tests:spmd_io_ops.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_iterator.mlir.test PASSED in 1.2s //tensorflow/dtensor/mlir/tests:spmd_matmul.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:spmd_random.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_save_restore.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_segment_sum.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:spmd_slice.mlir.test PASSED in 1.2s //tensorflow/dtensor/mlir/tests:spmd_softmax_loss.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:spmd_squeeze.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_var_handle.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:tf_dtensor_ops.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:tpu_add_resource_device_attribute.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:tpu_integration.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:undo_merge_const_across_mesh.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:update_tpu_metadata.mlir.test PASSED in 1.1s //tensorflow/dtensor/python/tests:collective_combine_all_reduce_test_cpu PASSED in 21.6s //tensorflow/dtensor/python/tests:collective_test_cpu PASSED in 17.4s //tensorflow/dtensor/python/tests:config_test_cpu PASSED in 9.9s //tensorflow/dtensor/python/tests:device_test_cpu PASSED in 43.1s //tensorflow/dtensor/python/tests:layout_test_cpu PASSED in 16.9s //tensorflow/dtensor/python/tests:multi_client_test_cpu PASSED in 17.7s //tensorflow/dtensor/python/tests:numpy_util_test_cpu PASSED in 10.2s //tensorflow/dtensor/tests:executable_manager_test PASSED in 42.3s //tensorflow/dtensor/tests:layout_to_xla_sharding_test PASSED in 0.5s //tensorflow/dtensor/tests:spmd_expander_test PASSED in 11.1s //tensorflow/dtensor/tests:tensor_layout_test PASSED in 2.0s //tensorflow/examples/adding_an_op:fact_test PASSED in 17.2s //tensorflow/examples/adding_an_op:zero_out_1_test PASSED in 19.5s //tensorflow/examples/adding_an_op:zero_out_2_test PASSED in 15.8s //tensorflow/examples/adding_an_op:zero_out_3_test PASSED in 15.1s //tensorflow/examples/custom_ops_doc/multiplex_1:multiplex_1_test PASSED in 16.2s //tensorflow/examples/custom_ops_doc/multiplex_2:multiplex_2_test_cpu PASSED in 17.2s //tensorflow/examples/custom_ops_doc/multiplex_3:multiplex_3_test PASSED in 16.5s //tensorflow/examples/custom_ops_doc/multiplex_4:multiplex_4_test PASSED in 15.3s //tensorflow/examples/custom_ops_doc/simple_hash_table:simple_hash_table_test PASSED in 19.5s //tensorflow/examples/custom_ops_doc/sleep:sleep_test PASSED in 28.0s //tensorflow/examples/speech_commands:accuracy_utils_test PASSED in 3.3s //tensorflow/examples/speech_commands:models_test PASSED in 17.8s //tensorflow/examples/speech_commands:recognize_commands_test PASSED in 2.9s //tensorflow/examples/wav_to_spectrogram:wav_to_spectrogram_test PASSED in 2.9s //tensorflow/js:ts_op_gen_test PASSED in 0.2s //tensorflow/python:array_grad_test_cpu PASSED in 9.5s //tensorflow/python:autograph_ops_test PASSED in 19.7s //tensorflow/python:batch_norm_benchmark_cpu PASSED in 6.9s //tensorflow/python:bincount_ops_test_cpu PASSED in 24.4s //tensorflow/python:bitwise_ops_test_cpu PASSED in 8.7s //tensorflow/python:clip_ops_test PASSED in 8.4s //tensorflow/python:clustering_ops_test PASSED in 24.4s //tensorflow/python:collective_ops_benchmark_cpu PASSED in 9.7s //tensorflow/python:collective_ops_gpu_test_2gpu PASSED in 12.2s //tensorflow/python:collective_ops_gpu_test_cpu PASSED in 9.6s //tensorflow/python:collective_ops_test PASSED in 17.3s //tensorflow/python:collective_ops_xla_test PASSED in 8.2s //tensorflow/python:compiled_collective_ops_gpu_test_2gpu PASSED in 7.2s //tensorflow/python:compiled_collective_ops_gpu_test_cpu PASSED in 10.7s //tensorflow/python:concat_benchmark_cpu PASSED in 6.8s //tensorflow/python:control_flow_ops_benchmark_cpu PASSED in 6.9s //tensorflow/python:control_flow_v2_enable_test PASSED in 7.5s //tensorflow/python:control_flow_v2_toggles_test PASSED in 8.6s //tensorflow/python:dequantize_op_test PASSED in 7.4s //tensorflow/python:embedding_ops_test_cpu PASSED in 11.2s //tensorflow/python:factory_ops_test_cpu PASSED in 7.9s //tensorflow/python:functional_ops_test PASSED in 7.9s //tensorflow/python:gradient_checker_v2_test_cpu PASSED in 28.0s //tensorflow/python:gradients_test_cpu PASSED in 19.3s //tensorflow/python:init_ops_test_cpu PASSED in 8.6s //tensorflow/python:init_ops_v2_test_cpu PASSED in 9.4s //tensorflow/python:math_grad_test_cpu PASSED in 17.8s //tensorflow/python:math_ops_linspace_test_cpu PASSED in 9.3s //tensorflow/python:math_ops_test_cpu PASSED in 33.9s //tensorflow/python:matmul_benchmark_cpu PASSED in 9.3s //tensorflow/python:nn_grad_test_cpu PASSED in 17.1s //tensorflow/python:nn_loss_scaling_utilities_test PASSED in 14.4s //tensorflow/python:nn_test_cpu PASSED in 60.4s //tensorflow/python:nn_xent_test_cpu PASSED in 10.0s //tensorflow/python:op_selector_test PASSED in 8.4s //tensorflow/python:ops/array_ops_shape_test PASSED in 6.2s //tensorflow/python:ops/array_ops_test PASSED in 7.5s //tensorflow/python:quantized_conv_ops_test PASSED in 9.5s //tensorflow/python:quantized_ops_test PASSED in 7.5s //tensorflow/python:raw_ops_test_cpu PASSED in 8.8s //tensorflow/python:rnn_grad_test_cpu PASSED in 8.1s //tensorflow/python:script_ops_test PASSED in 8.2s //tensorflow/python:sort_ops_test PASSED in 11.8s //tensorflow/python:sparse_ops_test PASSED in 17.9s //tensorflow/python:split_benchmark_cpu PASSED in 7.0s //tensorflow/python:tensor_array_ops_test PASSED in 9.0s //tensorflow/python:transpose_benchmark_cpu PASSED in 7.4s //tensorflow/python:variable_spec_test PASSED in 8.4s //tensorflow/python/autograph/converters:asserts_test PASSED in 11.6s //tensorflow/python/autograph/converters:break_statements_test PASSED in 7.7s //tensorflow/python/autograph/converters:call_trees_test PASSED in 10.9s //tensorflow/python/autograph/converters:conditional_expressions_test PASSED in 6.4s //tensorflow/python/autograph/converters:continue_statements_test PASSED in 9.3s //tensorflow/python/autograph/converters:control_flow_test PASSED in 15.9s //tensorflow/python/autograph/converters:directives_test PASSED in 9.2s //tensorflow/python/autograph/converters:functions_test PASSED in 7.6s //tensorflow/python/autograph/converters:list_comprehensions_test PASSED in 9.7s //tensorflow/python/autograph/converters:lists_test PASSED in 10.1s //tensorflow/python/autograph/converters:logical_expressions_test PASSED in 9.8s //tensorflow/python/autograph/converters:return_statements_test PASSED in 8.8s //tensorflow/python/autograph/converters:slices_test PASSED in 7.9s //tensorflow/python/autograph/converters:variables_test PASSED in 7.3s //tensorflow/python/autograph/core:converter_test PASSED in 6.7s //tensorflow/python/autograph/core:function_wrappers_test PASSED in 8.7s //tensorflow/python/autograph/impl:api_test PASSED in 14.0s //tensorflow/python/autograph/impl:conversion_test PASSED in 7.7s //tensorflow/python/autograph/lang:special_functions_test PASSED in 8.4s //tensorflow/python/autograph/operators:conditional_expressions_test PASSED in 9.2s //tensorflow/python/autograph/operators:control_flow_test PASSED in 15.0s //tensorflow/python/autograph/operators:data_structures_test PASSED in 8.9s //tensorflow/python/autograph/operators:exceptions_test PASSED in 7.9s //tensorflow/python/autograph/operators:logical_test PASSED in 8.4s //tensorflow/python/autograph/operators:py_builtins_test PASSED in 16.0s //tensorflow/python/autograph/operators:slices_test PASSED in 11.1s //tensorflow/python/autograph/operators:variables_test PASSED in 6.6s //tensorflow/python/autograph/pyct:anno_test PASSED in 9.1s //tensorflow/python/autograph/pyct:ast_util_test PASSED in 7.1s //tensorflow/python/autograph/pyct:cache_test PASSED in 6.3s //tensorflow/python/autograph/pyct:cfg_test PASSED in 8.3s //tensorflow/python/autograph/pyct:error_utils_test PASSED in 7.0s //tensorflow/python/autograph/pyct:inspect_utils_test PASSED in 10.3s //tensorflow/python/autograph/pyct:loader_test PASSED in 8.4s //tensorflow/python/autograph/pyct:naming_test PASSED in 6.5s //tensorflow/python/autograph/pyct:origin_info_test PASSED in 7.8s //tensorflow/python/autograph/pyct:parser_test PASSED in 9.5s //tensorflow/python/autograph/pyct:pretty_printer_test PASSED in 6.8s //tensorflow/python/autograph/pyct:qual_names_test PASSED in 6.4s //tensorflow/python/autograph/pyct:templates_test PASSED in 8.4s //tensorflow/python/autograph/pyct:transformer_test PASSED in 8.4s //tensorflow/python/autograph/pyct:transpiler_test PASSED in 8.5s //tensorflow/python/autograph/pyct/static_analysis:activity_test PASSED in 7.3s //tensorflow/python/autograph/pyct/static_analysis:liveness_test PASSED in 7.5s //tensorflow/python/autograph/pyct/static_analysis:reaching_definitions_test PASSED in 10.1s //tensorflow/python/autograph/pyct/static_analysis:reaching_fndefs_test PASSED in 7.2s //tensorflow/python/autograph/pyct/static_analysis:type_inference_test PASSED in 7.7s //tensorflow/python/autograph/tests:assertion_test PASSED in 17.8s //tensorflow/python/autograph/tests:basic_ifexp_test PASSED in 15.5s //tensorflow/python/autograph/tests:call_to_builtin_function_test PASSED in 28.4s //tensorflow/python/autograph/tests:call_to_lambda_function_test PASSED in 27.9s //tensorflow/python/autograph/tests:call_to_named_tuple_test PASSED in 27.3s //tensorflow/python/autograph/tests:call_to_numpy_function_test PASSED in 27.8s //tensorflow/python/autograph/tests:call_to_print_function_test PASSED in 15.2s //tensorflow/python/autograph/tests:call_to_tf_api_test PASSED in 14.8s //tensorflow/python/autograph/tests:call_to_user_function_test PASSED in 14.7s //tensorflow/python/autograph/tests:composite_names_in_control_flow_test PASSED in 33.1s //tensorflow/python/autograph/tests:cond_basic_test PASSED in 37.3s //tensorflow/python/autograph/tests:datasets_test PASSED in 28.7s //tensorflow/python/autograph/tests:early_return_test PASSED in 27.5s //tensorflow/python/autograph/tests:ext_slice_test PASSED in 15.0s //tensorflow/python/autograph/tests:generator_test PASSED in 15.8s //tensorflow/python/autograph/tests:logical_expression_test PASSED in 31.0s //tensorflow/python/autograph/tests:loop_basic_test PASSED in 107.9s //tensorflow/python/autograph/tests:loop_control_flow_illegal_cases_test PASSED in 29.8s //tensorflow/python/autograph/tests:loop_created_variables_test PASSED in 31.2s //tensorflow/python/autograph/tests:loop_scoping_test PASSED in 26.4s //tensorflow/python/autograph/tests:loop_with_function_call_test PASSED in 26.3s //tensorflow/python/autograph/tests:loop_with_variable_type_illegal_cases_test PASSED in 18.8s //tensorflow/python/autograph/tests:loop_with_variable_type_test PASSED in 48.6s //tensorflow/python/autograph/tests:nested_control_flow_test PASSED in 59.5s //tensorflow/python/autograph/tests:type_annotations_test PASSED in 14.3s //tensorflow/python/autograph/utils:context_managers_test PASSED in 7.4s //tensorflow/python/autograph/utils:misc_test PASSED in 7.1s //tensorflow/python/autograph/utils:tensor_list_test PASSED in 7.4s //tensorflow/python/autograph/utils:tensors_test PASSED in 7.0s //tensorflow/python/checkpoint:benchmarks_test PASSED in 7.4s //tensorflow/python/checkpoint:checkpoint_management_test_cpu PASSED in 19.6s //tensorflow/python/checkpoint:checkpoint_metrics_test PASSED in 15.0s //tensorflow/python/checkpoint:checkpoint_test PASSED in 37.1s //tensorflow/python/checkpoint:checkpoint_view_test PASSED in 9.0s //tensorflow/python/checkpoint:checkpoint_with_v1_optimizers_test PASSED in 10.6s //tensorflow/python/checkpoint:functional_saver_test_cpu PASSED in 22.9s //tensorflow/python/checkpoint:restore_test PASSED in 11.5s //tensorflow/python/checkpoint:save_util_v1_test PASSED in 8.0s //tensorflow/python/checkpoint:saveable_compat_test PASSED in 8.3s //tensorflow/python/checkpoint:tensor_callable_test PASSED in 13.3s //tensorflow/python/checkpoint:trackable_view_test PASSED in 7.1s //tensorflow/python/client:device_lib_test_cpu PASSED in 8.4s //tensorflow/python/client:events_writer_test PASSED in 7.1s //tensorflow/python/client:session_benchmark_cpu PASSED in 7.2s //tensorflow/python/client:session_list_devices_test PASSED in 10.5s //tensorflow/python/client:session_partial_run_test PASSED in 25.9s //tensorflow/python/client:timeline_test_cpu PASSED in 10.0s //tensorflow/python/client:virtual_gpu_test_cpu PASSED in 7.8s //tensorflow/python/compat:compat_test PASSED in 7.6s //tensorflow/python/compat:disable_v2_behavior_test PASSED in 11.7s //tensorflow/python/compiler/mlir:mlir_test PASSED in 7.1s //tensorflow/python/compiler/tensorrt:trt_convert_test_cpu PASSED in 10.0s //tensorflow/python/compiler/tensorrt/test:batch_matmul_test_cpu PASSED in 8.8s //tensorflow/python/compiler/tensorrt/test:biasadd_matmul_test_cpu PASSED in 7.0s //tensorflow/python/compiler/tensorrt/test:binary_tensor_weight_broadcast_test_cpu PASSED in 7.5s //tensorflow/python/compiler/tensorrt/test:bool_test_cpu PASSED in 12.1s //tensorflow/python/compiler/tensorrt/test:cast_test_cpu PASSED in 9.9s //tensorflow/python/compiler/tensorrt/test:concatenation_test_cpu PASSED in 11.3s //tensorflow/python/compiler/tensorrt/test:const_broadcast_test_cpu PASSED in 10.9s //tensorflow/python/compiler/tensorrt/test:data_dependent_shape_test_cpu PASSED in 7.8s //tensorflow/python/compiler/tensorrt/test:dynamic_input_shapes_test_cpu PASSED in 8.0s //tensorflow/python/compiler/tensorrt/test:identity_output_test_cpu PASSED in 9.3s //tensorflow/python/compiler/tensorrt/test:int32_test_cpu PASSED in 11.7s //tensorflow/python/compiler/tensorrt/test:lru_cache_test_cpu PASSED in 9.5s //tensorflow/python/compiler/tensorrt/test:memory_alignment_test_cpu PASSED in 8.6s //tensorflow/python/compiler/tensorrt/test:multi_connection_neighbor_engine_test_cpu PASSED in 10.4s //tensorflow/python/compiler/tensorrt/test:neighboring_engine_test_cpu PASSED in 9.8s //tensorflow/python/compiler/tensorrt/test:quantization_test_cpu PASSED in 8.4s //tensorflow/python/compiler/tensorrt/test:rank_two_test_cpu PASSED in 8.5s //tensorflow/python/compiler/tensorrt/test:reshape_transpose_test_cpu PASSED in 9.2s //tensorflow/python/compiler/tensorrt/test:topk_test_cpu PASSED in 8.4s //tensorflow/python/compiler/tensorrt/test:trt_engine_op_shape_test_cpu PASSED in 7.3s //tensorflow/python/compiler/tensorrt/test:trt_mode_test_cpu PASSED in 8.6s //tensorflow/python/compiler/tensorrt/test:unary_test_cpu PASSED in 8.2s //tensorflow/python/compiler/tensorrt/test:vgg_block_nchw_test_cpu PASSED in 9.3s //tensorflow/python/compiler/tensorrt/test:vgg_block_test_cpu PASSED in 7.8s //tensorflow/python/compiler/xla:jit_compile_test_cpu PASSED in 8.9s //tensorflow/python/compiler/xla:jit_test_cpu PASSED in 12.5s //tensorflow/python/compiler/xla:xla_test_cpu PASSED in 15.5s //tensorflow/python/compiler/xla/experimental:xla_sharding_test PASSED in 11.5s //tensorflow/python/data/benchmarks:batch_benchmark PASSED in 7.5s //tensorflow/python/data/benchmarks:filter_benchmark PASSED in 9.3s //tensorflow/python/data/benchmarks:from_tensor_slices_benchmark PASSED in 7.2s //tensorflow/python/data/benchmarks:interleave_benchmark PASSED in 7.3s //tensorflow/python/data/benchmarks:list_files_benchmark PASSED in 8.8s //tensorflow/python/data/benchmarks:map_benchmark PASSED in 8.1s //tensorflow/python/data/benchmarks:meta_benchmark PASSED in 8.8s //tensorflow/python/data/benchmarks:prefetch_benchmark PASSED in 7.7s //tensorflow/python/data/benchmarks:range_benchmark PASSED in 8.8s //tensorflow/python/data/experimental/benchmarks:autotune_benchmark PASSED in 6.9s //tensorflow/python/data/experimental/benchmarks:csv_dataset_benchmark PASSED in 9.2s //tensorflow/python/data/experimental/benchmarks:map_and_batch_benchmark PASSED in 9.0s //tensorflow/python/data/experimental/benchmarks:map_defun_benchmark PASSED in 9.2s //tensorflow/python/data/experimental/benchmarks:matching_files_benchmark PASSED in 8.2s //tensorflow/python/data/experimental/benchmarks:optimize_benchmark PASSED in 7.6s //tensorflow/python/data/experimental/benchmarks:parameter_value_benchmark PASSED in 7.9s //tensorflow/python/data/experimental/benchmarks:rejection_resample_benchmark PASSED in 8.0s //tensorflow/python/data/experimental/benchmarks:snapshot_dataset_benchmark PASSED in 6.9s //tensorflow/python/data/experimental/benchmarks:unbatch_benchmark PASSED in 8.0s //tensorflow/python/data/experimental/kernel_tests:assert_cardinality_test PASSED in 38.2s //tensorflow/python/data/experimental/kernel_tests:assert_next_test PASSED in 11.1s //tensorflow/python/data/experimental/kernel_tests:assert_prev_test PASSED in 13.0s //tensorflow/python/data/experimental/kernel_tests:checkpoint_input_pipeline_hook_test PASSED in 18.2s //tensorflow/python/data/experimental/kernel_tests:compression_ops_test PASSED in 21.6s //tensorflow/python/data/experimental/kernel_tests:copy_to_device_test_cpu PASSED in 16.0s //tensorflow/python/data/experimental/kernel_tests:dense_to_sparse_batch_test PASSED in 23.8s //tensorflow/python/data/experimental/kernel_tests:from_list_test PASSED in 49.8s //tensorflow/python/data/experimental/kernel_tests:io_test PASSED in 95.5s //tensorflow/python/data/experimental/kernel_tests:lookup_ops_test PASSED in 22.3s //tensorflow/python/data/experimental/kernel_tests:make_csv_dataset_test PASSED in 48.1s //tensorflow/python/data/experimental/kernel_tests:make_saveable_from_iterator_test PASSED in 7.4s //tensorflow/python/data/experimental/kernel_tests:make_tf_record_dataset_test PASSED in 87.7s //tensorflow/python/data/experimental/kernel_tests:map_defun_op_test PASSED in 7.4s //tensorflow/python/data/experimental/kernel_tests:matching_files_dataset_test PASSED in 16.2s //tensorflow/python/data/experimental/kernel_tests:model_dataset_test PASSED in 10.8s //tensorflow/python/data/experimental/kernel_tests:non_serializable_test PASSED in 9.5s //tensorflow/python/data/experimental/kernel_tests:pad_to_cardinality_test PASSED in 10.7s //tensorflow/python/data/experimental/kernel_tests:prefetch_to_device_test_cpu PASSED in 12.3s //tensorflow/python/data/experimental/kernel_tests:prefetch_with_slack_test PASSED in 11.4s //tensorflow/python/data/experimental/kernel_tests:shuffle_and_repeat_test PASSED in 23.1s //tensorflow/python/data/experimental/kernel_tests:sleep_test PASSED in 7.2s //tensorflow/python/data/experimental/kernel_tests:tf_record_writer_test PASSED in 20.0s //tensorflow/python/data/experimental/kernel_tests:variant_test PASSED in 8.3s //tensorflow/python/data/experimental/kernel_tests:wrap_unwrap_test_cpu PASSED in 19.8s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_fusion_test PASSED in 36.2s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_parallelization_test PASSED in 78.6s //tensorflow/python/data/experimental/kernel_tests/optimization:grappler_test_cpu PASSED in 9.1s //tensorflow/python/data/experimental/kernel_tests/optimization:make_deterministic_test PASSED in 36.2s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_batch_fusion_test PASSED in 9.4s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_filter_fusion_test PASSED in 28.7s //tensorflow/python/data/experimental/kernel_tests/optimization:map_fusion_test PASSED in 31.4s //tensorflow/python/data/experimental/kernel_tests/optimization:map_parallelization_test PASSED in 14.0s //tensorflow/python/data/experimental/kernel_tests/optimization:noop_elimination_test PASSED in 13.9s //tensorflow/python/data/experimental/kernel_tests/service:multi_device_test PASSED in 15.1s //tensorflow/python/data/experimental/service:server_lib_test PASSED in 9.4s //tensorflow/python/data/kernel_tests:as_numpy_iterator_test PASSED in 8.9s //tensorflow/python/data/kernel_tests:bucket_by_sequence_length_test PASSED in 19.8s //tensorflow/python/data/kernel_tests:cache_test PASSED in 79.9s //tensorflow/python/data/kernel_tests:cardinality_test PASSED in 13.8s //tensorflow/python/data/kernel_tests:checkpoint_test PASSED in 29.1s //tensorflow/python/data/kernel_tests:concatenate_test PASSED in 23.1s //tensorflow/python/data/kernel_tests:counter_test PASSED in 31.3s //tensorflow/python/data/kernel_tests:dataset_spec_test PASSED in 7.6s //tensorflow/python/data/kernel_tests:dataset_test PASSED in 40.1s //tensorflow/python/data/kernel_tests:enumerate_test PASSED in 46.0s //tensorflow/python/data/kernel_tests:from_sparse_tensor_slices_test PASSED in 7.4s //tensorflow/python/data/kernel_tests:from_tensor_slices_test PASSED in 27.5s //tensorflow/python/data/kernel_tests:from_tensors_test PASSED in 20.9s //tensorflow/python/data/kernel_tests:get_single_element_test PASSED in 12.2s //tensorflow/python/data/kernel_tests:ignore_errors_test PASSED in 28.2s //tensorflow/python/data/kernel_tests:io_test PASSED in 93.5s //tensorflow/python/data/kernel_tests:iterator_test_cpu PASSED in 32.2s //tensorflow/python/data/kernel_tests:len_test PASSED in 7.6s //tensorflow/python/data/kernel_tests:list_files_test PASSED in 13.8s //tensorflow/python/data/kernel_tests:optional_test_cpu PASSED in 12.4s //tensorflow/python/data/kernel_tests:options_test PASSED in 10.3s //tensorflow/python/data/kernel_tests:placement_test_cpu PASSED in 10.6s //tensorflow/python/data/kernel_tests:prefetch_test PASSED in 61.4s //tensorflow/python/data/kernel_tests:random_test PASSED in 32.4s //tensorflow/python/data/kernel_tests:range_test PASSED in 55.0s //tensorflow/python/data/kernel_tests:rebatch_test PASSED in 7.4s //tensorflow/python/data/kernel_tests:reduce_test_cpu PASSED in 24.3s //tensorflow/python/data/kernel_tests:scan_test_cpu PASSED in 72.4s //tensorflow/python/data/kernel_tests:sparse_batch_test PASSED in 19.8s //tensorflow/python/data/kernel_tests:unbatch_test PASSED in 55.5s //tensorflow/python/data/util:convert_test PASSED in 7.3s //tensorflow/python/data/util:nest_test PASSED in 8.3s //tensorflow/python/data/util:options_test PASSED in 9.9s //tensorflow/python/data/util:random_seed_test PASSED in 7.7s //tensorflow/python/data/util:sparse_test PASSED in 9.5s //tensorflow/python/data/util:structure_test PASSED in 9.9s //tensorflow/python/data/util:traverse_test PASSED in 6.9s //tensorflow/python/debug/cli:analyzer_cli_test_cpu PASSED in 11.0s //tensorflow/python/debug/cli:cli_config_test PASSED in 9.9s //tensorflow/python/debug/cli:cli_shared_test PASSED in 7.8s //tensorflow/python/debug/cli:command_parser_test PASSED in 9.1s //tensorflow/python/debug/cli:curses_ui_test PASSED in 6.8s //tensorflow/python/debug/cli:debugger_cli_common_test PASSED in 6.7s //tensorflow/python/debug/cli:evaluator_test PASSED in 7.7s //tensorflow/python/debug/cli:profile_analyzer_cli_test PASSED in 10.2s //tensorflow/python/debug/cli:readline_ui_test PASSED in 11.0s //tensorflow/python/debug/cli:tensor_format_test PASSED in 8.3s //tensorflow/python/debug/lib:check_numerics_callback_test_cpu PASSED in 13.7s //tensorflow/python/debug/lib:common_test PASSED in 7.2s //tensorflow/python/debug/lib:debug_data_test PASSED in 8.4s //tensorflow/python/debug/lib:debug_events_monitors_test PASSED in 10.2s //tensorflow/python/debug/lib:debug_events_writer_test PASSED in 13.3s //tensorflow/python/debug/lib:debug_gradients_test_cpu PASSED in 8.1s //tensorflow/python/debug/lib:debug_graph_reconstruction_test_cpu PASSED in 14.2s //tensorflow/python/debug/lib:debug_graphs_test PASSED in 7.4s //tensorflow/python/debug/lib:debug_grappler_test_cpu PASSED in 8.6s //tensorflow/python/debug/lib:debug_utils_test PASSED in 7.0s //tensorflow/python/debug/lib:debug_v2_ops_test_cpu PASSED in 22.7s //tensorflow/python/debug/lib:profiling_test PASSED in 8.3s //tensorflow/python/debug/lib:session_debug_file_test_cpu PASSED in 16.2s //tensorflow/python/debug/lib:session_debug_multi_gpu_test_cpu PASSED in 10.1s //tensorflow/python/debug/lib:source_utils_test PASSED in 14.7s //tensorflow/python/debug/wrappers:disk_usage_test PASSED in 10.1s //tensorflow/python/debug/wrappers:dumping_wrapper_test PASSED in 7.8s //tensorflow/python/debug/wrappers:framework_test PASSED in 7.5s //tensorflow/python/debug/wrappers:local_cli_wrapper_test PASSED in 7.7s //tensorflow/python/distribute:checkpoint_utils_test_2gpu PASSED in 10.4s //tensorflow/python/distribute:checkpoint_utils_test_cpu PASSED in 11.3s //tensorflow/python/distribute:checkpointing_test_2gpu PASSED in 8.0s //tensorflow/python/distribute:checkpointing_test_cpu PASSED in 12.3s //tensorflow/python/distribute:collective_util_test PASSED in 7.0s //tensorflow/python/distribute:combinations_test_2gpu PASSED in 23.4s //tensorflow/python/distribute:combinations_test_cpu PASSED in 22.8s //tensorflow/python/distribute:cross_device_utils_test_cpu PASSED in 8.2s //tensorflow/python/distribute:custom_training_loop_gradient_test_2gpu PASSED in 10.3s //tensorflow/python/distribute:custom_training_loop_gradient_test_cpu PASSED in 14.2s //tensorflow/python/distribute:device_util_test_cpu PASSED in 10.4s //tensorflow/python/distribute:distribute_coordinator_test PASSED in 14.0s //tensorflow/python/distribute:distribute_lib_test PASSED in 14.0s //tensorflow/python/distribute:distribute_utils_test_2gpu PASSED in 8.9s //tensorflow/python/distribute:distribute_utils_test_cpu PASSED in 9.8s //tensorflow/python/distribute:input_ops_test_cpu PASSED in 14.1s //tensorflow/python/distribute:metrics_v1_test_2gpu PASSED in 31.4s //tensorflow/python/distribute:metrics_v1_test_cpu PASSED in 50.7s //tensorflow/python/distribute:mirrored_values_test_2gpu PASSED in 9.6s //tensorflow/python/distribute:mirrored_values_test_cpu PASSED in 8.9s //tensorflow/python/distribute:mirrored_variable_test_2gpu PASSED in 27.7s //tensorflow/python/distribute:mirrored_variable_test_cpu PASSED in 20.5s //tensorflow/python/distribute:multi_process_runner_no_init_test PASSED in 15.1s //tensorflow/python/distribute:multi_worker_continuous_run_test_cpu PASSED in 25.8s //tensorflow/python/distribute:multi_worker_util_test PASSED in 8.5s //tensorflow/python/distribute:numpy_dataset_test PASSED in 8.8s //tensorflow/python/distribute:one_device_strategy_test_cpu PASSED in 33.3s //tensorflow/python/distribute:packed_distributed_variable_test PASSED in 9.1s //tensorflow/python/distribute:parameter_server_strategy_test_2gpu PASSED in 42.7s //tensorflow/python/distribute:parameter_server_strategy_test_cpu PASSED in 31.0s //tensorflow/python/distribute:parameter_server_strategy_v2_test_2gpu PASSED in 29.2s //tensorflow/python/distribute:parameter_server_strategy_v2_test_cpu PASSED in 19.8s //tensorflow/python/distribute:per_replica_test_2gpu PASSED in 9.4s //tensorflow/python/distribute:per_replica_test_cpu PASSED in 10.6s //tensorflow/python/distribute:ps_values_test_2gpu PASSED in 17.1s //tensorflow/python/distribute:ps_values_test_cpu PASSED in 10.2s //tensorflow/python/distribute:remote_mirrored_strategy_eager_test_cpu PASSED in 9.9s //tensorflow/python/distribute:sharded_variable_test PASSED in 29.0s //tensorflow/python/distribute:shared_variable_creator_test PASSED in 8.9s //tensorflow/python/distribute:strategy_combinations_test_cpu PASSED in 45.7s //tensorflow/python/distribute:template_mirrored_strategy_test_cpu PASSED in 14.8s //tensorflow/python/distribute:test_util_test_2gpu PASSED in 17.9s //tensorflow/python/distribute:test_util_test_cpu PASSED in 24.3s //tensorflow/python/distribute:tf_function_test_2gpu PASSED in 12.5s //tensorflow/python/distribute:tf_function_test_cpu PASSED in 13.7s //tensorflow/python/distribute:values_v2_test_cpu PASSED in 12.5s //tensorflow/python/distribute:warm_starting_util_test_2gpu PASSED in 9.3s //tensorflow/python/distribute:warm_starting_util_test_cpu PASSED in 11.7s //tensorflow/python/distribute/cluster_resolver:base_cluster_resolver_py_test PASSED in 11.9s //tensorflow/python/distribute/cluster_resolver:gce_cluster_resolver_py_test PASSED in 8.6s //tensorflow/python/distribute/cluster_resolver:kubernetes_cluster_resolver_py_test PASSED in 7.5s //tensorflow/python/distribute/cluster_resolver:sagemaker_cluster_resolver_py_test PASSED in 7.3s //tensorflow/python/distribute/cluster_resolver:slurm_cluster_resolver_py_test PASSED in 7.3s //tensorflow/python/distribute/cluster_resolver:tfconfig_cluster_resolver_py_test PASSED in 11.6s //tensorflow/python/distribute/cluster_resolver/tpu:tpu_cluster_resolver_py_test PASSED in 12.0s //tensorflow/python/distribute/coordinator:metric_utils_test PASSED in 13.4s //tensorflow/python/distribute/coordinator:watchdog_test PASSED in 62.4s //tensorflow/python/distribute/experimental:dtensor_util_test_cpu PASSED in 13.2s //tensorflow/python/distribute/experimental:mirrored_strategy_test_cpu PASSED in 42.2s //tensorflow/python/distribute/experimental:multi_worker_mirrored_strategy_test_cpu PASSED in 28.0s //tensorflow/python/distribute/integration_test:saved_model_test_cpu PASSED in 46.7s //tensorflow/python/distribute/parallel_device:parallel_device_test_cpu PASSED in 11.1s //tensorflow/python/distribute/v1:all_reduce_test PASSED in 48.6s //tensorflow/python/distribute/v1:cross_device_ops_test_2gpu PASSED in 64.8s //tensorflow/python/distribute/v1:cross_device_ops_test_cpu PASSED in 78.3s //tensorflow/python/dlpack:dlpack_test_cpu PASSED in 11.2s //tensorflow/python/eager:backprop_test_cpu PASSED in 130.3s //tensorflow/python/eager:benchmarks_test_cpu PASSED in 9.6s //tensorflow/python/eager:cancellation_test_cpu PASSED in 9.6s //tensorflow/python/eager:context_test_cpu PASSED in 9.2s //tensorflow/python/eager:core_test_cpu PASSED in 22.8s //tensorflow/python/eager:gradient_input_output_exclusions_test PASSED in 34.8s //tensorflow/python/eager:graph_only_ops_test_cpu PASSED in 7.6s //tensorflow/python/eager:lift_to_graph_test PASSED in 8.7s //tensorflow/python/eager:monitoring_test_cpu PASSED in 10.2s //tensorflow/python/eager:ops_test_cpu PASSED in 10.4s //tensorflow/python/eager:profiler_client_test PASSED in 8.1s //tensorflow/python/eager:profiler_test_cpu PASSED in 6.7s //tensorflow/python/eager:pywrap_tfe_test PASSED in 23.7s //tensorflow/python/eager:record_test PASSED in 19.3s //tensorflow/python/eager:remote_benchmarks_test_cpu PASSED in 7.4s //tensorflow/python/eager:run_eager_op_as_function_test_cpu PASSED in 6.9s //tensorflow/python/eager:run_eager_op_as_function_xla_test_cpu PASSED in 9.2s //tensorflow/python/eager:small_constants_optimizer_test_cpu PASSED in 188.3s //tensorflow/python/eager:tensor_test_cpu PASSED in 15.3s //tensorflow/python/eager:wrap_function_device_test_cpu PASSED in 9.3s //tensorflow/python/eager:wrap_function_test PASSED in 10.2s //tensorflow/python/eager/benchmarks:kpi_benchmark_test_cpu PASSED in 12.2s //tensorflow/python/eager/memory_tests:remote_memory_test_cpu PASSED in 10.0s //tensorflow/python/eager/polymorphic_function:argument_naming_test_cpu PASSED in 10.5s //tensorflow/python/eager/polymorphic_function:atomic_function_test_cpu PASSED in 12.3s //tensorflow/python/eager/polymorphic_function:collection_test_cpu PASSED in 8.6s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu PASSED in 7.1s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu_mlir_bridge_test PASSED in 10.3s //tensorflow/python/eager/polymorphic_function:function_spec_test PASSED in 7.3s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_jit_test_cpu PASSED in 32.4s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_jit_test_cpu_mlir_bridge_test PASSED in 29.9s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_test_cpu PASSED in 8.0s //tensorflow/python/eager/polymorphic_function:quarantine_test PASSED in 34.8s //tensorflow/python/feature_column:sequence_feature_column_integration_test PASSED in 9.7s //tensorflow/python/feature_column:serialization_test PASSED in 14.2s //tensorflow/python/framework:auto_control_deps_test PASSED in 25.4s //tensorflow/python/framework:c_api_util_test PASSED in 10.1s //tensorflow/python/framework:common_shapes_test PASSED in 12.0s //tensorflow/python/framework:composite_tensor_test PASSED in 9.6s //tensorflow/python/framework:config_test_2gpu PASSED in 11.5s //tensorflow/python/framework:config_test_cpu PASSED in 12.0s //tensorflow/python/framework:constant_op_test PASSED in 15.1s //tensorflow/python/framework:device_spec_test PASSED in 7.4s //tensorflow/python/framework:device_test PASSED in 6.4s //tensorflow/python/framework:dtypes_test PASSED in 17.7s //tensorflow/python/framework:error_interpolation_test PASSED in 9.7s //tensorflow/python/framework:errors_test PASSED in 7.7s //tensorflow/python/framework:extension_type_field_test PASSED in 9.9s //tensorflow/python/framework:extension_type_test PASSED in 20.0s //tensorflow/python/framework:file_system_test PASSED in 7.4s //tensorflow/python/framework:function_def_to_graph_test PASSED in 8.4s //tensorflow/python/framework:graph_building_benchmark_cpu PASSED in 19.9s //tensorflow/python/framework:graph_util_test PASSED in 7.7s //tensorflow/python/framework:immutable_dict_test PASSED in 6.8s //tensorflow/python/framework:importer_test PASSED in 9.5s //tensorflow/python/framework:indexed_slices_test PASSED in 7.6s //tensorflow/python/framework:kernels_test PASSED in 7.2s //tensorflow/python/framework:meta_graph_test PASSED in 10.0s //tensorflow/python/framework:node_file_writer_test_cpu PASSED in 7.8s //tensorflow/python/framework:offset_counter_helper_test PASSED in 0.6s //tensorflow/python/framework:op_allowlist_namespace_test PASSED in 1.9s //tensorflow/python/framework:op_callbacks_test_cpu PASSED in 13.5s //tensorflow/python/framework:op_def_library_test PASSED in 9.4s //tensorflow/python/framework:op_def_util_test PASSED in 8.0s //tensorflow/python/framework:ops_enable_eager_test PASSED in 3.3s //tensorflow/python/framework:ops_test PASSED in 30.1s //tensorflow/python/framework:proto_test PASSED in 10.2s //tensorflow/python/framework:py_context_manager_test PASSED in 7.4s //tensorflow/python/framework:python_api_dispatcher_test PASSED in 7.5s //tensorflow/python/framework:python_api_info_test PASSED in 11.2s //tensorflow/python/framework:python_api_parameter_converter_test PASSED in 8.3s //tensorflow/python/framework:python_op_gen_annotation_test PASSED in 3.4s //tensorflow/python/framework:python_op_gen_annotator_test PASSED in 0.3s //tensorflow/python/framework:python_tensor_converter_test PASSED in 7.9s //tensorflow/python/framework:random_seed_test PASSED in 10.9s //tensorflow/python/framework:registry_test PASSED in 11.8s //tensorflow/python/framework:smart_cond_test PASSED in 10.5s //tensorflow/python/framework:sparse_tensor_test PASSED in 7.9s //tensorflow/python/framework:subscribe_test PASSED in 8.0s //tensorflow/python/framework:tensor_shape_test PASSED in 7.3s //tensorflow/python/framework:tensor_test PASSED in 8.5s //tensorflow/python/framework:tensor_util_test PASSED in 10.4s //tensorflow/python/framework:test_combinations_test PASSED in 6.7s //tensorflow/python/framework:test_util_test_cpu PASSED in 16.6s //tensorflow/python/framework:tf2_test PASSED in 11.6s //tensorflow/python/framework:traceable_stack_test PASSED in 6.9s //tensorflow/python/framework:type_spec_test PASSED in 7.7s //tensorflow/python/framework:versions_test PASSED in 7.3s //tensorflow/python/framework/experimental:graph_building_test_cpu PASSED in 8.4s //tensorflow/python/framework/experimental:unified_api_test_cpu PASSED in 14.3s //tensorflow/python/grappler:arithmetic_optimizer_test_cpu PASSED in 7.4s //tensorflow/python/grappler:auto_mixed_precision_test_cpu PASSED in 11.4s //tensorflow/python/grappler:constant_folding_test_cpu PASSED in 13.2s //tensorflow/python/grappler:cost_analyzer_test PASSED in 9.3s //tensorflow/python/grappler:datasets_test PASSED in 8.2s //tensorflow/python/grappler:item_test PASSED in 8.3s //tensorflow/python/grappler:memory_optimizer_test PASSED in 16.0s //tensorflow/python/grappler:model_analyzer_test PASSED in 8.2s //tensorflow/python/grappler:remapper_test_cpu PASSED in 8.5s //tensorflow/python/grappler:tf_optimizer_test PASSED in 6.9s //tensorflow/python/kernel_tests:benchmark_test_cpu PASSED in 10.7s //tensorflow/python/kernel_tests:check_ops_test_cpu PASSED in 17.6s //tensorflow/python/kernel_tests:collective_ops_multi_worker_test PASSED in 36.1s //tensorflow/python/kernel_tests:composite_tensor_ops_test PASSED in 14.2s //tensorflow/python/kernel_tests:critical_section_test_cpu PASSED in 22.3s //tensorflow/python/kernel_tests:garbage_collection_test PASSED in 8.6s //tensorflow/python/kernel_tests:gradient_correctness_test_cpu PASSED in 7.7s //tensorflow/python/kernel_tests:histogram_ops_test_cpu PASSED in 9.5s //tensorflow/python/kernel_tests:logging_ops_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests:numerics_test_cpu PASSED in 8.5s //tensorflow/python/kernel_tests:template_test PASSED in 10.9s //tensorflow/python/kernel_tests:trace_op_test_cpu PASSED in 7.5s //tensorflow/python/kernel_tests/array_ops:batch_gather_op_test_cpu PASSED in 10.7s //tensorflow/python/kernel_tests/array_ops:batch_scatter_ops_test PASSED in 8.8s //tensorflow/python/kernel_tests/array_ops:batchtospace_op_test_cpu PASSED in 14.9s //tensorflow/python/kernel_tests/array_ops:bcast_ops_test PASSED in 8.7s //tensorflow/python/kernel_tests/array_ops:bitcast_op_test_cpu PASSED in 8.8s //tensorflow/python/kernel_tests/array_ops:broadcast_to_ops_test_cpu PASSED in 40.6s //tensorflow/python/kernel_tests/array_ops:cast_op_test_cpu PASSED in 9.9s //tensorflow/python/kernel_tests/array_ops:constant_op_eager_test_cpu PASSED in 13.9s //tensorflow/python/kernel_tests/array_ops:constant_op_test_cpu PASSED in 12.5s //tensorflow/python/kernel_tests/array_ops:denormal_test_cpu PASSED in 7.7s //tensorflow/python/kernel_tests/array_ops:depthtospace_op_test_cpu PASSED in 10.6s //tensorflow/python/kernel_tests/array_ops:edit_distance_op_test PASSED in 9.2s //tensorflow/python/kernel_tests/array_ops:fingerprint_op_test PASSED in 9.1s //tensorflow/python/kernel_tests/array_ops:gather_nd_op_test_cpu PASSED in 7.9s //tensorflow/python/kernel_tests/array_ops:identity_n_op_py_test PASSED in 7.0s //tensorflow/python/kernel_tests/array_ops:identity_op_py_test PASSED in 9.4s //tensorflow/python/kernel_tests/array_ops:large_concat_op_test_cpu PASSED in 8.7s //tensorflow/python/kernel_tests/array_ops:manip_ops_test_cpu PASSED in 11.4s //tensorflow/python/kernel_tests/array_ops:one_hot_op_test_cpu PASSED in 8.8s //tensorflow/python/kernel_tests/array_ops:pad_op_test_cpu PASSED in 17.0s //tensorflow/python/kernel_tests/array_ops:reshape_op_test_cpu PASSED in 7.6s //tensorflow/python/kernel_tests/array_ops:reverse_sequence_op_test_cpu PASSED in 8.1s //tensorflow/python/kernel_tests/array_ops:scalar_test_cpu PASSED in 9.3s //tensorflow/python/kernel_tests/array_ops:shape_ops_test_cpu PASSED in 14.5s //tensorflow/python/kernel_tests/array_ops:slice_op_test_cpu PASSED in 13.2s //tensorflow/python/kernel_tests/array_ops:spacetobatch_op_test_cpu PASSED in 14.0s //tensorflow/python/kernel_tests/array_ops:spacetodepth_op_test_cpu PASSED in 8.4s //tensorflow/python/kernel_tests/array_ops:stack_op_test_cpu PASSED in 16.9s //tensorflow/python/kernel_tests/array_ops:unique_op_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests/array_ops:unstack_op_test_cpu PASSED in 9.8s //tensorflow/python/kernel_tests/array_ops:where_op_test_cpu PASSED in 17.9s //tensorflow/python/kernel_tests/control_flow:cond_v2_test_cpu PASSED in 53.5s //tensorflow/python/kernel_tests/control_flow:control_flow_util_test PASSED in 9.7s //tensorflow/python/kernel_tests/control_flow:control_flow_util_v2_test PASSED in 9.8s //tensorflow/python/kernel_tests/control_flow:py_func_test_cpu PASSED in 17.1s //tensorflow/python/kernel_tests/control_flow:scan_ops_test_cpu PASSED in 84.7s //tensorflow/python/kernel_tests/control_flow:while_v2_test_cpu PASSED in 58.2s //tensorflow/python/kernel_tests/custom_ops:ackermann_test PASSED in 7.6s //tensorflow/python/kernel_tests/custom_ops:duplicate_op_test PASSED in 9.1s //tensorflow/python/kernel_tests/custom_ops:invalid_op_test PASSED in 8.5s //tensorflow/python/kernel_tests/data_structures:conditional_accumulator_test PASSED in 9.2s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_2gpu PASSED in 17.5s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_cpu PASSED in 15.4s //tensorflow/python/kernel_tests/data_structures:dynamic_stitch_op_test_cpu PASSED in 10.4s //tensorflow/python/kernel_tests/data_structures:fifo_queue_test PASSED in 19.5s //tensorflow/python/kernel_tests/data_structures:list_ops_test_cpu PASSED in 22.9s //tensorflow/python/kernel_tests/data_structures:listdiff_op_test PASSED in 7.6s //tensorflow/python/kernel_tests/data_structures:lookup_ops_test PASSED in 27.1s //tensorflow/python/kernel_tests/data_structures:map_ops_test PASSED in 13.4s //tensorflow/python/kernel_tests/data_structures:padding_fifo_queue_test_cpu PASSED in 7.4s //tensorflow/python/kernel_tests/data_structures:priority_queue_test PASSED in 7.4s //tensorflow/python/kernel_tests/data_structures:stack_ops_test_cpu PASSED in 8.4s //tensorflow/python/kernel_tests/data_structures:stage_op_test_cpu PASSED in 8.8s //tensorflow/python/kernel_tests/distributions:bernoulli_test_cpu PASSED in 18.3s //tensorflow/python/kernel_tests/distributions:bijector_test_cpu PASSED in 12.7s //tensorflow/python/kernel_tests/distributions:categorical_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/distributions:dirichlet_multinomial_test_cpu PASSED in 16.3s //tensorflow/python/kernel_tests/distributions:dirichlet_test_cpu PASSED in 16.0s //tensorflow/python/kernel_tests/distributions:exponential_test_cpu PASSED in 14.5s //tensorflow/python/kernel_tests/distributions:gamma_test_cpu PASSED in 54.4s //tensorflow/python/kernel_tests/distributions:identity_bijector_test_cpu PASSED in 8.2s //tensorflow/python/kernel_tests/distributions:kullback_leibler_test_cpu PASSED in 19.9s //tensorflow/python/kernel_tests/distributions:laplace_test_cpu PASSED in 38.2s //tensorflow/python/kernel_tests/distributions:multinomial_test_cpu PASSED in 9.7s //tensorflow/python/kernel_tests/distributions:normal_test_cpu PASSED in 28.8s //tensorflow/python/kernel_tests/distributions:special_math_test_cpu PASSED in 32.6s //tensorflow/python/kernel_tests/distributions:uniform_test_cpu PASSED in 17.9s //tensorflow/python/kernel_tests/image_ops:attention_ops_test PASSED in 20.7s //tensorflow/python/kernel_tests/image_ops:decode_bmp_op_test PASSED in 20.3s //tensorflow/python/kernel_tests/image_ops:decode_compressed_op_test PASSED in 8.0s //tensorflow/python/kernel_tests/image_ops:decode_image_op_test PASSED in 8.3s //tensorflow/python/kernel_tests/image_ops:decode_jpeg_op_test PASSED in 7.9s //tensorflow/python/kernel_tests/image_ops:decode_png_op_test PASSED in 7.9s //tensorflow/python/kernel_tests/image_ops:decode_raw_op_test PASSED in 6.6s //tensorflow/python/kernel_tests/image_ops:draw_bounding_box_op_test_cpu PASSED in 9.0s //tensorflow/python/kernel_tests/image_ops:extract_image_patches_op_test_cpu PASSED in 7.3s //tensorflow/python/kernel_tests/image_ops:extract_volume_patches_op_test_cpu PASSED in 6.9s //tensorflow/python/kernel_tests/io_ops:checkpoint_ops_test PASSED in 18.3s //tensorflow/python/kernel_tests/io_ops:decode_csv_op_test PASSED in 10.2s //tensorflow/python/kernel_tests/io_ops:io_ops_test PASSED in 7.4s //tensorflow/python/kernel_tests/io_ops:parse_single_example_op_test PASSED in 18.0s //tensorflow/python/kernel_tests/io_ops:parsing_ops_test PASSED in 28.3s //tensorflow/python/kernel_tests/io_ops:reader_ops_test PASSED in 14.4s //tensorflow/python/kernel_tests/io_ops:record_input_test PASSED in 50.1s //tensorflow/python/kernel_tests/io_ops:save_restore_ops_test PASSED in 9.3s //tensorflow/python/kernel_tests/linalg:determinant_op_test_cpu PASSED in 6.8s //tensorflow/python/kernel_tests/linalg:linear_operator_addition_test_cpu PASSED in 8.4s //tensorflow/python/kernel_tests/linalg:linear_operator_algebra_test_cpu PASSED in 7.6s //tensorflow/python/kernel_tests/linalg:linear_operator_test_cpu PASSED in 13.6s //tensorflow/python/kernel_tests/linalg:lu_op_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests/linalg:matrix_inverse_op_test_cpu PASSED in 9.9s //tensorflow/python/kernel_tests/linalg:matrix_logarithm_op_test PASSED in 59.4s //tensorflow/python/kernel_tests/linalg:matrix_solve_ls_op_test_cpu PASSED in 58.1s //tensorflow/python/kernel_tests/linalg:matrix_solve_op_test_cpu PASSED in 34.9s //tensorflow/python/kernel_tests/linalg:matrix_square_root_op_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/linalg:slicing_test_cpu PASSED in 24.6s //tensorflow/python/kernel_tests/linalg/sparse:conjugate_gradient_test_cpu PASSED in 13.5s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_test_cpu PASSED in 10.5s //tensorflow/python/kernel_tests/math_ops:aggregate_ops_test_cpu PASSED in 12.3s //tensorflow/python/kernel_tests/math_ops:argmax_op_test_cpu PASSED in 9.6s //tensorflow/python/kernel_tests/math_ops:banded_triangular_solve_op_test_cpu PASSED in 18.9s //tensorflow/python/kernel_tests/math_ops:basic_gpu_test_cpu PASSED in 7.4s //tensorflow/python/kernel_tests/math_ops:bincount_op_test_cpu PASSED in 10.4s //tensorflow/python/kernel_tests/math_ops:bucketize_op_test_cpu PASSED in 8.2s //tensorflow/python/kernel_tests/math_ops:clip_ops_test PASSED in 15.2s //tensorflow/python/kernel_tests/math_ops:confusion_matrix_test PASSED in 11.8s //tensorflow/python/kernel_tests/math_ops:cross_grad_test_cpu PASSED in 9.5s //tensorflow/python/kernel_tests/math_ops:cumulative_logsumexp_test_cpu PASSED in 8.4s //tensorflow/python/kernel_tests/math_ops:in_topk_op_test_cpu PASSED in 7.5s //tensorflow/python/kernel_tests/math_ops:reduce_benchmark_test_cpu PASSED in 10.4s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_d9m_test_cpu PASSED in 13.6s //tensorflow/python/kernel_tests/math_ops:sets_test PASSED in 26.7s //tensorflow/python/kernel_tests/math_ops:topk_op_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests/math_ops:zero_division_test_cpu PASSED in 8.1s //tensorflow/python/kernel_tests/nn_ops:betainc_op_test_cpu PASSED in 9.4s //tensorflow/python/kernel_tests/nn_ops:bias_op_test_cpu PASSED in 172.9s //tensorflow/python/kernel_tests/nn_ops:conv1d_test_cpu PASSED in 7.7s //tensorflow/python/kernel_tests/nn_ops:conv1d_transpose_test_cpu PASSED in 7.0s //tensorflow/python/kernel_tests/nn_ops:conv2d_transpose_test_cpu PASSED in 8.4s //tensorflow/python/kernel_tests/nn_ops:conv3d_backprop_filter_v2_grad_test_cpu PASSED in 22.5s //tensorflow/python/kernel_tests/nn_ops:conv3d_transpose_test_cpu PASSED in 8.2s //tensorflow/python/kernel_tests/nn_ops:ctc_decoder_ops_test PASSED in 9.8s //tensorflow/python/kernel_tests/nn_ops:ctc_loss_op_test_cpu PASSED in 78.1s //tensorflow/python/kernel_tests/nn_ops:cudnn_d9m_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/nn_ops:cudnn_deterministic_ops_test_cpu PASSED in 7.6s //tensorflow/python/kernel_tests/nn_ops:losses_test PASSED in 35.7s //tensorflow/python/kernel_tests/nn_ops:lrn_op_test_cpu PASSED in 8.9s //tensorflow/python/kernel_tests/nn_ops:morphological_ops_test_cpu PASSED in 13.5s //tensorflow/python/kernel_tests/nn_ops:nth_element_op_test_cpu PASSED in 11.4s //tensorflow/python/kernel_tests/nn_ops:pool_test_cpu PASSED in 49.1s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_3d_test_cpu PASSED in 35.4s //tensorflow/python/kernel_tests/nn_ops:relu_op_test_cpu PASSED in 9.9s //tensorflow/python/kernel_tests/nn_ops:softmax_op_test_cpu PASSED in 9.7s //tensorflow/python/kernel_tests/nn_ops:softplus_op_test_cpu PASSED in 11.5s //tensorflow/python/kernel_tests/nn_ops:softsign_op_test_cpu PASSED in 7.2s //tensorflow/python/kernel_tests/nn_ops:xent_op_d9m_test_cpu PASSED in 154.6s //tensorflow/python/kernel_tests/nn_ops:xent_op_test_cpu PASSED in 9.4s //tensorflow/python/kernel_tests/proto:descriptor_source_test PASSED in 7.0s //tensorflow/python/kernel_tests/proto:encode_proto_op_test PASSED in 7.8s //tensorflow/python/kernel_tests/quantization_ops:quantization_ops_test PASSED in 8.2s //tensorflow/python/kernel_tests/random:candidate_sampler_ops_test PASSED in 7.8s //tensorflow/python/kernel_tests/random:multinomial_op_test_cpu PASSED in 8.7s //tensorflow/python/kernel_tests/random:parameterized_truncated_normal_op_test_cpu PASSED in 15.7s //tensorflow/python/kernel_tests/random:random_crop_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests/random:random_grad_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests/random:random_ops_test_cpu PASSED in 14.8s //tensorflow/python/kernel_tests/random:random_poisson_test_cpu PASSED in 12.1s //tensorflow/python/kernel_tests/random:random_shuffle_queue_test PASSED in 6.8s //tensorflow/python/kernel_tests/random:stateful_random_ops_test_cpu PASSED in 18.3s //tensorflow/python/kernel_tests/signal:mel_ops_test_cpu PASSED in 14.1s //tensorflow/python/kernel_tests/signal:mfcc_ops_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/signal:reconstruction_ops_test_cpu PASSED in 11.6s //tensorflow/python/kernel_tests/signal:shape_ops_test_cpu PASSED in 27.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_add_op_test PASSED in 10.1s //tensorflow/python/kernel_tests/sparse_ops:sparse_concat_op_test PASSED in 8.2s //tensorflow/python/kernel_tests/sparse_ops:sparse_conditional_accumulator_test PASSED in 10.7s //tensorflow/python/kernel_tests/sparse_ops:sparse_cross_op_test PASSED in 14.1s //tensorflow/python/kernel_tests/sparse_ops:sparse_matmul_op_test_cpu PASSED in 39.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_reorder_op_test PASSED in 8.7s //tensorflow/python/kernel_tests/sparse_ops:sparse_reshape_op_test PASSED in 10.5s //tensorflow/python/kernel_tests/sparse_ops:sparse_serialization_ops_test PASSED in 11.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_slice_op_test PASSED in 10.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_split_op_test_cpu PASSED in 7.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_grad_test_cpu PASSED in 19.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_d9m_test_cpu PASSED in 39.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_test_cpu PASSED in 44.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensors_map_ops_test PASSED in 9.1s //tensorflow/python/kernel_tests/sparse_ops:sparse_to_dense_op_py_test_cpu PASSED in 11.5s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_d9m_test_cpu PASSED in 88.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_test_cpu PASSED in 9.6s //tensorflow/python/kernel_tests/sparse_ops:sparsemask_op_test PASSED in 8.9s //tensorflow/python/kernel_tests/strings_ops:as_string_op_test PASSED in 10.3s //tensorflow/python/kernel_tests/strings_ops:base64_ops_test PASSED in 12.2s //tensorflow/python/kernel_tests/strings_ops:reduce_join_op_test_cpu PASSED in 11.6s //tensorflow/python/kernel_tests/strings_ops:regex_full_match_op_test PASSED in 8.1s //tensorflow/python/kernel_tests/strings_ops:regex_replace_op_test PASSED in 8.0s //tensorflow/python/kernel_tests/strings_ops:string_bytes_split_op_test PASSED in 20.8s //tensorflow/python/kernel_tests/strings_ops:string_format_op_test PASSED in 8.6s //tensorflow/python/kernel_tests/strings_ops:string_join_op_test PASSED in 11.5s //tensorflow/python/kernel_tests/strings_ops:string_length_op_test PASSED in 7.3s //tensorflow/python/kernel_tests/strings_ops:string_lower_op_test PASSED in 8.1s //tensorflow/python/kernel_tests/strings_ops:string_split_op_test PASSED in 22.7s //tensorflow/python/kernel_tests/strings_ops:string_strip_op_test PASSED in 7.3s //tensorflow/python/kernel_tests/strings_ops:string_to_hash_bucket_op_test_cpu PASSED in 8.1s //tensorflow/python/kernel_tests/strings_ops:string_to_number_op_test_cpu PASSED in 7.9s //tensorflow/python/kernel_tests/strings_ops:string_upper_op_test PASSED in 9.8s //tensorflow/python/kernel_tests/strings_ops:substr_op_test PASSED in 9.6s //tensorflow/python/kernel_tests/strings_ops:unicode_decode_op_test PASSED in 17.4s //tensorflow/python/kernel_tests/strings_ops:unicode_encode_op_test PASSED in 21.0s //tensorflow/python/kernel_tests/strings_ops:unicode_script_op_test PASSED in 8.0s //tensorflow/python/kernel_tests/strings_ops:unicode_transcode_op_test PASSED in 10.3s //tensorflow/python/kernel_tests/strings_ops:unsorted_segment_join_op_test_cpu PASSED in 9.7s //tensorflow/python/kernel_tests/summary_ops:summary_ops_test_cpu PASSED in 18.9s //tensorflow/python/kernel_tests/summary_ops:summary_v1_audio_op_test_cpu PASSED in 7.7s //tensorflow/python/kernel_tests/summary_ops:summary_v1_image_op_test_cpu PASSED in 9.7s //tensorflow/python/kernel_tests/summary_ops:summary_v1_ops_test PASSED in 8.7s //tensorflow/python/kernel_tests/summary_ops:summary_v1_tensor_op_test PASSED in 8.0s //tensorflow/python/kernel_tests/v1_compat_tests:array_ops_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/v1_compat_tests:dense_update_ops_test_cpu PASSED in 7.1s //tensorflow/python/kernel_tests/v1_compat_tests:identity_op_py_test PASSED in 7.5s //tensorflow/python/kernel_tests/v1_compat_tests:scatter_nd_ops_test_cpu PASSED in 7.4s //tensorflow/python/kernel_tests/v1_compat_tests:session_ops_test_cpu PASSED in 9.4s //tensorflow/python/kernel_tests/v1_compat_tests:stack_op_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/variables:dense_update_ops_no_tsan_test_cpu PASSED in 8.6s //tensorflow/python/kernel_tests/variables:dense_update_ops_test_cpu PASSED in 7.9s //tensorflow/python/kernel_tests/variables:partitioned_variables_test PASSED in 16.0s //tensorflow/python/kernel_tests/variables:resource_variable_ops_test_cpu PASSED in 57.7s //tensorflow/python/kernel_tests/variables:variable_ops_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests/variables:variable_scope_test PASSED in 37.6s //tensorflow/python/kernel_tests/variables:variables_test PASSED in 9.1s //tensorflow/python/lib/core:custom_float_test PASSED in 9.9s //tensorflow/python/lib/io:file_io_test PASSED in 9.4s //tensorflow/python/lib/io:tf_record_test PASSED in 8.6s //tensorflow/python/module:module_test PASSED in 8.4s //tensorflow/python/ops/losses:util_test PASSED in 7.8s //tensorflow/python/ops/memory_tests:custom_gradient_memory_test_cpu PASSED in 15.2s //tensorflow/python/ops/numpy_ops:np_array_ops_test_cpu PASSED in 82.9s //tensorflow/python/ops/numpy_ops:np_arrays_test_cpu PASSED in 10.0s //tensorflow/python/ops/numpy_ops:np_dtypes_test_cpu PASSED in 6.6s //tensorflow/python/ops/numpy_ops:np_interop_test_cpu PASSED in 41.3s //tensorflow/python/ops/numpy_ops:np_logic_test_cpu PASSED in 11.7s //tensorflow/python/ops/numpy_ops:np_math_ops_test_cpu PASSED in 26.7s //tensorflow/python/ops/numpy_ops:np_random_test_cpu PASSED in 74.1s //tensorflow/python/ops/numpy_ops:np_utils_test_cpu PASSED in 12.4s //tensorflow/python/ops/numpy_ops/integration_test:np_config_test_cpu PASSED in 15.7s //tensorflow/python/ops/numpy_ops/integration_test:public_symbol_test PASSED in 15.6s //tensorflow/python/ops/parallel_for:array_test_cpu PASSED in 48.3s //tensorflow/python/ops/parallel_for:gradients_test_cpu PASSED in 12.7s //tensorflow/python/ops/parallel_for:xla_control_flow_ops_test_cpu PASSED in 65.1s //tensorflow/python/ops/ragged:convert_to_tensor_or_ragged_tensor_op_test PASSED in 9.3s //tensorflow/python/ops/ragged:ragged_batch_gather_op_test PASSED in 46.8s //tensorflow/python/ops/ragged:ragged_bitcast_op_test PASSED in 7.8s //tensorflow/python/ops/ragged:ragged_boolean_mask_op_test PASSED in 20.4s //tensorflow/python/ops/ragged:ragged_concat_op_test PASSED in 10.7s //tensorflow/python/ops/ragged:ragged_const_op_test PASSED in 8.3s //tensorflow/python/ops/ragged:ragged_constant_value_op_test PASSED in 7.4s //tensorflow/python/ops/ragged:ragged_cross_op_test PASSED in 31.3s //tensorflow/python/ops/ragged:ragged_dispatch_test PASSED in 145.5s //tensorflow/python/ops/ragged:ragged_dynamic_partition_op_test_cpu PASSED in 19.1s //tensorflow/python/ops/ragged:ragged_eager_test PASSED in 10.1s //tensorflow/python/ops/ragged:ragged_expand_dims_op_test PASSED in 8.9s //tensorflow/python/ops/ragged:ragged_factory_ops_test_cpu PASSED in 14.8s //tensorflow/python/ops/ragged:ragged_fill_empty_rows_op_test PASSED in 14.5s //tensorflow/python/ops/ragged:ragged_from_sparse_op_test PASSED in 10.5s //tensorflow/python/ops/ragged:ragged_from_tensor_op_test PASSED in 25.4s //tensorflow/python/ops/ragged:ragged_gather_nd_op_test PASSED in 9.7s //tensorflow/python/ops/ragged:ragged_map_flat_values_op_test PASSED in 11.6s //tensorflow/python/ops/ragged:ragged_map_fn_op_test PASSED in 19.6s //tensorflow/python/ops/ragged:ragged_math_ops_test PASSED in 17.1s //tensorflow/python/ops/ragged:ragged_matmul_op_test PASSED in 49.7s //tensorflow/python/ops/ragged:ragged_merge_dims_op_test PASSED in 37.3s //tensorflow/python/ops/ragged:ragged_one_hot_op_test PASSED in 12.2s //tensorflow/python/ops/ragged:ragged_operators_test PASSED in 29.7s //tensorflow/python/ops/ragged:ragged_placeholder_op_test PASSED in 6.6s //tensorflow/python/ops/ragged:ragged_print_op_test PASSED in 15.6s //tensorflow/python/ops/ragged:ragged_range_op_test PASSED in 10.0s //tensorflow/python/ops/ragged:ragged_rank_op_test PASSED in 7.9s //tensorflow/python/ops/ragged:ragged_reduce_op_test PASSED in 42.5s //tensorflow/python/ops/ragged:ragged_resize_image_op_test PASSED in 25.8s //tensorflow/python/ops/ragged:ragged_reverse_op_test PASSED in 7.8s //tensorflow/python/ops/ragged:ragged_row_lengths_op_test PASSED in 7.3s //tensorflow/python/ops/ragged:ragged_row_splits_to_segment_ids_op_test PASSED in 14.2s //tensorflow/python/ops/ragged:ragged_segment_ids_to_row_splits_op_test PASSED in 6.9s //tensorflow/python/ops/ragged:ragged_segment_op_test PASSED in 23.8s //tensorflow/python/ops/ragged:ragged_size_op_test PASSED in 9.3s //tensorflow/python/ops/ragged:ragged_split_op_test PASSED in 51.6s //tensorflow/python/ops/ragged:ragged_squeeze_op_test PASSED in 17.3s //tensorflow/python/ops/ragged:ragged_stack_op_test PASSED in 19.9s //tensorflow/python/ops/ragged:ragged_tensor_bounding_shape_op_test PASSED in 12.0s //tensorflow/python/ops/ragged:ragged_tensor_shape_test PASSED in 66.8s //tensorflow/python/ops/ragged:ragged_tile_op_test PASSED in 51.9s //tensorflow/python/ops/ragged:ragged_to_sparse_op_test PASSED in 8.7s //tensorflow/python/ops/ragged:ragged_to_tensor_op_test PASSED in 80.4s //tensorflow/python/ops/ragged:ragged_util_test PASSED in 31.2s //tensorflow/python/ops/ragged:ragged_where_op_test PASSED in 42.6s //tensorflow/python/ops/ragged:row_partition_test PASSED in 39.6s //tensorflow/python/ops/ragged:string_ngrams_op_test PASSED in 9.3s //tensorflow/python/ops/ragged:strings_reduce_join_op_test PASSED in 9.0s //tensorflow/python/ops/structured:structured_array_ops_test PASSED in 46.6s //tensorflow/python/ops/structured:structured_tensor_slice_test PASSED in 69.4s //tensorflow/python/ops/structured:structured_tensor_spec_test PASSED in 10.6s //tensorflow/python/ops/structured:structured_tensor_test PASSED in 45.6s //tensorflow/python/ops/v1_compat_tests:gradient_checker_test_cpu PASSED in 13.3s //tensorflow/python/platform:benchmark_test PASSED in 9.5s //tensorflow/python/platform:build_info_test PASSED in 7.6s //tensorflow/python/platform:resource_loader_test PASSED in 2.0s //tensorflow/python/profiler:pprof_profiler_test PASSED in 8.5s //tensorflow/python/profiler:profile_context_test_cpu PASSED in 35.1s //tensorflow/python/profiler:profiler_client_test_cpu PASSED in 8.4s //tensorflow/python/profiler:profiler_test_cpu PASSED in 20.0s //tensorflow/python/profiler:profiler_v2_test_cpu PASSED in 12.6s //tensorflow/python/profiler:profiler_wrapper_test PASSED in 10.0s //tensorflow/python/profiler:tfprof_logger_test PASSED in 7.2s //tensorflow/python/profiler/internal:flops_registry_test PASSED in 9.1s //tensorflow/python/profiler/internal:print_model_analysis_test PASSED in 7.1s //tensorflow/python/profiler/internal:run_metadata_test_cpu PASSED in 13.9s //tensorflow/python/saved_model:fingerprinting_test PASSED in 15.2s //tensorflow/python/saved_model:keras_injection_test PASSED in 14.9s //tensorflow/python/saved_model:load_v1_in_v2_test PASSED in 29.4s //tensorflow/python/saved_model:loader_test PASSED in 15.5s //tensorflow/python/saved_model:method_name_updater_test PASSED in 9.9s //tensorflow/python/saved_model:metrics_test PASSED in 12.7s //tensorflow/python/saved_model:nested_structure_coder_test PASSED in 7.6s //tensorflow/python/saved_model:pywrap_saved_model_fingerprinting_test PASSED in 7.8s //tensorflow/python/saved_model:pywrap_saved_model_metrics_test PASSED in 7.7s //tensorflow/python/saved_model:revived_types_test PASSED in 8.9s //tensorflow/python/saved_model:save_context_test PASSED in 9.9s //tensorflow/python/saved_model:save_test PASSED in 39.8s //tensorflow/python/saved_model:saved_model_test PASSED in 18.5s //tensorflow/python/saved_model:signature_def_utils_test PASSED in 7.7s //tensorflow/python/saved_model:simple_save_test PASSED in 8.4s //tensorflow/python/saved_model:tracing_utils_test PASSED in 10.5s //tensorflow/python/saved_model:utils_test PASSED in 8.0s //tensorflow/python/saved_model/model_utils:export_output_test PASSED in 7.8s //tensorflow/python/saved_model/model_utils:export_test PASSED in 15.0s //tensorflow/python/saved_model/model_utils:mode_keys_test PASSED in 11.8s //tensorflow/python/saved_model/registration:registration_saving_test PASSED in 15.2s //tensorflow/python/saved_model/registration:registration_test PASSED in 8.8s //tensorflow/python/saved_model/registration:tf_registration_test PASSED in 19.3s //tensorflow/python/summary:plugin_asset_test PASSED in 9.0s //tensorflow/python/summary:summary_iterator_test PASSED in 6.6s //tensorflow/python/summary:summary_test PASSED in 8.4s //tensorflow/python/summary:summary_v2_test PASSED in 8.4s //tensorflow/python/summary/writer:writer_test PASSED in 20.4s //tensorflow/python/tools:aot_compiled_test PASSED in 27.6s //tensorflow/python/tools:freeze_graph_test PASSED in 16.5s //tensorflow/python/tools:optimize_for_inference_test PASSED in 9.7s //tensorflow/python/tools:print_selective_registration_header_test PASSED in 13.7s //tensorflow/python/tools:saved_model_cli_test PASSED in 28.1s //tensorflow/python/tools:saved_model_utils_test PASSED in 9.3s //tensorflow/python/tools:strip_unused_test PASSED in 6.5s //tensorflow/python/tools/api/generator:create_python_api_test PASSED in 12.8s //tensorflow/python/tools/api/generator:output_init_files_test PASSED in 14.0s //tensorflow/python/tools/api/generator:tensorflow_doc_srcs_test PASSED in 11.5s //tensorflow/python/tpu:bfloat16_test PASSED in 7.5s //tensorflow/python/tpu:feature_column_test PASSED in 10.9s //tensorflow/python/tpu:topology_test PASSED in 8.8s //tensorflow/python/tpu:tpu_embedding_for_serving_test PASSED in 9.5s //tensorflow/python/tpu:tpu_embedding_v2_utils_test PASSED in 9.3s //tensorflow/python/tpu:tpu_infeed_test PASSED in 9.5s //tensorflow/python/tpu:tpu_sharding_test PASSED in 7.0s //tensorflow/python/tpu:tpu_test_wrapper_test PASSED in 7.8s //tensorflow/python/tpu/client:client_py_test PASSED in 9.6s //tensorflow/python/trackable:autotrackable_test PASSED in 9.0s //tensorflow/python/trackable:base_delegate_test PASSED in 7.8s //tensorflow/python/trackable:base_test PASSED in 8.3s //tensorflow/python/trackable:data_structures_test PASSED in 18.9s //tensorflow/python/trackable:python_state_test PASSED in 9.0s //tensorflow/python/trackable:resource_test PASSED in 6.8s //tensorflow/python/trackable:trackable_utils_test PASSED in 8.7s //tensorflow/python/training:adadelta_test_cpu PASSED in 18.6s //tensorflow/python/training:adagrad_da_test_cpu PASSED in 9.7s //tensorflow/python/training:adagrad_test_cpu PASSED in 16.7s //tensorflow/python/training:adam_test_cpu PASSED in 16.1s //tensorflow/python/training:basic_loops_test_cpu PASSED in 9.1s //tensorflow/python/training:basic_session_run_hooks_test PASSED in 20.4s //tensorflow/python/training:checkpoint_ops_test PASSED in 7.8s //tensorflow/python/training:coordinator_test_cpu PASSED in 18.6s //tensorflow/python/training:device_setter_test_cpu PASSED in 7.8s //tensorflow/python/training:ftrl_test_cpu PASSED in 19.5s //tensorflow/python/training:gradient_descent_test_cpu PASSED in 10.7s //tensorflow/python/training:input_test PASSED in 22.8s //tensorflow/python/training:momentum_test_cpu PASSED in 12.9s //tensorflow/python/training:monitored_session_test PASSED in 25.5s //tensorflow/python/training:moving_averages_test_cpu PASSED in 18.7s //tensorflow/python/training:optimizer_test_cpu PASSED in 13.9s //tensorflow/python/training:proximal_adagrad_test_cpu PASSED in 9.7s //tensorflow/python/training:proximal_gradient_descent_test_cpu PASSED in 8.8s //tensorflow/python/training:quantize_training_test_cpu PASSED in 7.1s //tensorflow/python/training:queue_runner_test_cpu PASSED in 7.8s //tensorflow/python/training:rmsprop_test_cpu PASSED in 33.6s //tensorflow/python/training:saver_large_partitioned_variable_test PASSED in 16.4s //tensorflow/python/training:saver_test_2gpu PASSED in 33.7s //tensorflow/python/training:saver_test_cpu PASSED in 40.5s //tensorflow/python/training:server_lib_multiple_containers_test PASSED in 7.8s //tensorflow/python/training:server_lib_same_variables_clear_container_test PASSED in 8.6s //tensorflow/python/training:server_lib_same_variables_clear_test PASSED in 10.1s //tensorflow/python/training:server_lib_same_variables_no_clear_test PASSED in 7.7s //tensorflow/python/training:server_lib_sparse_job_test PASSED in 9.1s //tensorflow/python/training:server_lib_test PASSED in 20.5s //tensorflow/python/training:session_manager_test_cpu PASSED in 77.6s //tensorflow/python/training:slot_creator_test_cpu PASSED in 7.8s //tensorflow/python/training:supervisor_test PASSED in 16.6s //tensorflow/python/training:training_ops_mlir_test_cpu PASSED in 10.1s //tensorflow/python/training:training_ops_test_cpu PASSED in 9.4s //tensorflow/python/training:training_util_test PASSED in 8.3s //tensorflow/python/training:warm_starting_util_test PASSED in 25.1s //tensorflow/python/training/experimental:loss_scale_optimizer_test PASSED in 18.6s //tensorflow/python/training/experimental:loss_scale_test PASSED in 27.1s //tensorflow/python/training/experimental:mixed_precision_test_cpu PASSED in 9.9s //tensorflow/python/training/saving:saveable_object_util_test PASSED in 7.9s //tensorflow/python/util:compat_test PASSED in 8.4s //tensorflow/python/util:decorator_utils_test PASSED in 7.0s //tensorflow/python/util:deprecation_test PASSED in 8.2s //tensorflow/python/util:dispatch_test PASSED in 13.6s //tensorflow/python/util:example_parser_configuration_test PASSED in 7.2s //tensorflow/python/util:fast_module_type_test PASSED in 7.2s //tensorflow/python/util:function_parameter_canonicalizer_test PASSED in 7.0s //tensorflow/python/util:function_utils_test PASSED in 6.6s //tensorflow/python/util:keyword_args_test PASSED in 7.4s //tensorflow/python/util:lazy_loader_test PASSED in 11.3s //tensorflow/python/util:lock_util_test PASSED in 9.7s //tensorflow/python/util:module_wrapper_test PASSED in 7.6s //tensorflow/python/util:nest_test PASSED in 13.9s //tensorflow/python/util:object_identity_test PASSED in 11.6s //tensorflow/python/util:pywrap_xla_ops_test PASSED in 2.2s //tensorflow/python/util:serialization_test PASSED in 7.0s //tensorflow/python/util:tf_contextlib_test PASSED in 6.9s //tensorflow/python/util:tf_decorator_test PASSED in 6.8s //tensorflow/python/util:tf_export_test PASSED in 7.8s //tensorflow/python/util:tf_inspect_test PASSED in 7.3s //tensorflow/python/util:tf_should_use_test PASSED in 7.3s //tensorflow/python/util:tf_stack_test PASSED in 9.9s //tensorflow/python/util:traceback_utils_test PASSED in 7.5s //tensorflow/python/util:type_annotations_test PASSED in 7.2s //tensorflow/python/util:variable_utils_test PASSED in 7.8s //tensorflow/python/util:vlog_test PASSED in 7.7s //tensorflow/tools/api/tests:module_test PASSED in 26.3s //tensorflow/tools/benchmark:benchmark_model_test PASSED in 2.2s //tensorflow/tools/common:public_api_test PASSED in 2.9s //tensorflow/tools/common:traverse_test PASSED in 2.6s //tensorflow/tools/compatibility:all_renames_v2_test PASSED in 6.2s //tensorflow/tools/compatibility:ast_edits_test PASSED in 6.3s //tensorflow/tools/compatibility:test_file_v1_0 PASSED in 14.6s //tensorflow/tools/compatibility:test_file_v2_0 PASSED in 15.5s //tensorflow/tools/compatibility:tf_upgrade_test PASSED in 8.0s //tensorflow/tools/compatibility:tf_upgrade_v2_safety_test PASSED in 7.6s //tensorflow/tools/docs:tf_doctest_test PASSED in 0.8s //tensorflow/tools/graph_transforms:file_utils_test PASSED in 1.3s //tensorflow/tools/graph_transforms:transform_graph_test PASSED in 2.0s //tensorflow/tools/graph_transforms:transform_utils_test PASSED in 2.6s //tensorflow/tools/graph_transforms:transforms_test PASSED in 4.2s //tensorflow/tools/proto_text:gen_proto_text_functions_lib_test PASSED in 0.3s //tensorflow/tools/tensorflow_builder/compat_checker:compat_checker_test PASSED in 0.3s //tensorflow/tsl/c:tsl_status_helper_test PASSED in 0.2s //tensorflow/tsl/c:tsl_status_test PASSED in 0.1s //tensorflow/tsl/concurrency:async_value_ref_test PASSED in 1.0s //tensorflow/tsl/concurrency:async_value_test PASSED in 1.0s //tensorflow/tsl/concurrency:concurrent_vector_test PASSED in 0.5s //tensorflow/tsl/cuda:cudnn_version_test PASSED in 0.2s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_agent_test PASSED in 12.9s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_error_util_test PASSED in 0.5s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_recoverable_job_test PASSED in 0.8s //tensorflow/tsl/distributed_runtime/persistent_cache_repository:fake_cache_repository_test PASSED in 0.6s //tensorflow/tsl/distributed_runtime/preemption:preemption_notifier_test PASSED in 5.8s //tensorflow/tsl/distributed_runtime/preemption:preemption_sync_manager_test PASSED in 6.2s //tensorflow/tsl/distributed_runtime/rpc:grpc_channel_test PASSED in 0.7s //tensorflow/tsl/distributed_runtime/rpc:grpc_util_test PASSED in 0.3s //tensorflow/tsl/framework:cancellation_test PASSED in 1.8s //tensorflow/tsl/framework:device_id_utils_test PASSED in 4.0s //tensorflow/tsl/framework/convolution:spatial_convolutions_test PASSED in 0.2s //tensorflow/tsl/lib/gtl:tsl_lib_gtl_tests PASSED in 0.7s //tensorflow/tsl/lib/hash:crc32c_test PASSED in 0.1s //tensorflow/tsl/lib/histogram:histogram_test PASSED in 0.5s //tensorflow/tsl/lib/io:buffered_inputstream_test PASSED in 0.5s //tensorflow/tsl/lib/io:cache_test PASSED in 0.1s //tensorflow/tsl/lib/io:inputbuffer_test PASSED in 1.7s //tensorflow/tsl/lib/io:inputstream_interface_test PASSED in 0.4s //tensorflow/tsl/lib/io:random_inputstream_test PASSED in 0.3s //tensorflow/tsl/lib/io:record_reader_writer_test PASSED in 1.2s //tensorflow/tsl/lib/io:recordio_test PASSED in 1.2s //tensorflow/tsl/lib/io:table_test PASSED in 4.3s //tensorflow/tsl/lib/io:zlib_buffers_test PASSED in 8.3s //tensorflow/tsl/lib/io/snappy:snappy_test PASSED in 0.8s //tensorflow/tsl/lib/math:math_util_test PASSED in 0.6s //tensorflow/tsl/lib/random:distribution_sampler_test PASSED in 0.3s //tensorflow/tsl/lib/random:philox_random_test PASSED in 0.1s //tensorflow/tsl/lib/random:random_distributions_test PASSED in 17.5s //tensorflow/tsl/lib/random:simple_philox_test PASSED in 0.3s //tensorflow/tsl/lib/random:weighted_picker_test PASSED in 12.6s //tensorflow/tsl/platform:ctstring_test PASSED in 0.3s //tensorflow/tsl/platform:denormal_test PASSED in 0.4s //tensorflow/tsl/platform:errors_test PASSED in 0.2s //tensorflow/tsl/platform:fingerprint_test PASSED in 0.1s //tensorflow/tsl/platform:float8_test PASSED in 1.2s //tensorflow/tsl/platform:hash_test PASSED in 0.1s //tensorflow/tsl/platform:integral_types_test PASSED in 0.9s //tensorflow/tsl/platform:intrusive_ptr_test PASSED in 0.4s //tensorflow/tsl/platform:logging_test PASSED in 21.5s //tensorflow/tsl/platform:mutex_test PASSED in 0.5s //tensorflow/tsl/platform:net_test PASSED in 0.1s //tensorflow/tsl/platform:numbers_test PASSED in 0.6s //tensorflow/tsl/platform:path_test PASSED in 0.1s //tensorflow/tsl/platform:port_test PASSED in 8.6s //tensorflow/tsl/platform:random_test PASSED in 1.8s //tensorflow/tsl/platform:refcount_test PASSED in 0.7s //tensorflow/tsl/platform:retrying_file_system_test PASSED in 1.0s //tensorflow/tsl/platform:retrying_utils_test PASSED in 0.1s //tensorflow/tsl/platform:scanner_test PASSED in 0.5s //tensorflow/tsl/platform:setround_test PASSED in 0.2s //tensorflow/tsl/platform:stacktrace_handler_test PASSED in 2.3s //tensorflow/tsl/platform:stacktrace_test PASSED in 0.5s //tensorflow/tsl/platform:status_matchers_test PASSED in 0.7s //tensorflow/tsl/platform:status_test PASSED in 0.2s //tensorflow/tsl/platform:statusor_test PASSED in 2.1s //tensorflow/tsl/platform:str_util_test PASSED in 0.6s //tensorflow/tsl/platform:strcat_test PASSED in 0.8s //tensorflow/tsl/platform:stringpiece_test PASSED in 0.3s //tensorflow/tsl/platform:stringprintf_test PASSED in 0.4s //tensorflow/tsl/platform:subprocess_test PASSED in 0.9s //tensorflow/tsl/platform:tstring_test PASSED in 0.3s //tensorflow/tsl/platform:unbounded_work_queue_test PASSED in 1.1s //tensorflow/tsl/platform/cloud:compute_engine_metadata_client_test PASSED in 1.3s //tensorflow/tsl/platform/cloud:compute_engine_zone_provider_test PASSED in 0.2s //tensorflow/tsl/platform/cloud:curl_http_request_test PASSED in 9.6s //tensorflow/tsl/platform/cloud:expiring_lru_cache_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:gcs_dns_cache_test PASSED in 0.3s //tensorflow/tsl/platform/cloud:gcs_file_system_test PASSED in 7.2s //tensorflow/tsl/platform/cloud:gcs_throttle_test PASSED in 0.4s //tensorflow/tsl/platform/cloud:google_auth_provider_test PASSED in 0.2s //tensorflow/tsl/platform/cloud:oauth_client_test PASSED in 0.3s //tensorflow/tsl/platform/cloud:ram_file_block_cache_test PASSED in 2.5s //tensorflow/tsl/platform/cloud:time_util_test PASSED in 0.6s //tensorflow/tsl/profiler/backends/cpu:traceme_recorder_test PASSED in 0.6s //tensorflow/tsl/profiler/convert:trace_container_test PASSED in 0.2s //tensorflow/tsl/profiler/convert:trace_events_to_json_test PASSED in 0.1s //tensorflow/tsl/profiler/convert:xla_op_utils_test PASSED in 0.1s //tensorflow/tsl/profiler/convert:xplane_to_trace_events_test PASSED in 0.7s //tensorflow/tsl/profiler/lib:profiler_factory_test PASSED in 0.3s //tensorflow/tsl/profiler/lib:profiler_lock_test PASSED in 0.4s //tensorflow/tsl/profiler/lib:scoped_annotation_test PASSED in 0.2s //tensorflow/tsl/profiler/lib:traceme_encode_test PASSED in 0.1s //tensorflow/tsl/profiler/rpc/client:profiler_client_test PASSED in 3.6s //tensorflow/tsl/profiler/rpc/client:remote_profiler_session_manager_test PASSED in 3.7s //tensorflow/tsl/profiler/utils:buffer_pool_test PASSED in 0.5s //tensorflow/tsl/profiler/utils:group_events_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:parse_annotation_test PASSED in 0.2s //tensorflow/tsl/profiler/utils:preprocess_xplane_test PASSED in 0.3s //tensorflow/tsl/profiler/utils:tf_op_utils_test PASSED in 0.7s //tensorflow/tsl/profiler/utils:timespan_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:tpu_xplane_utils_test PASSED in 0.4s //tensorflow/tsl/profiler/utils:xplane_builder_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:xplane_utils_test PASSED in 0.2s //tensorflow/tsl/util:device_name_utils_test PASSED in 0.2s //tensorflow/tsl/util:stats_calculator_test PASSED in 0.1s //tensorflow/compiler/tests:complex_div_test_cpu PASSED in 7.3s Stats over 2 runs: max = 7.3s, min = 6.5s, avg = 6.9s, dev = 0.4s //tensorflow/compiler/tests:complex_div_test_cpu_mlir_bridge_test PASSED in 9.9s Stats over 2 runs: max = 9.9s, min = 7.5s, avg = 8.7s, dev = 1.2s //tensorflow/compiler/xla/tests:conditional_test_cpu PASSED in 18.4s Stats over 2 runs: max = 18.4s, min = 13.0s, avg = 15.7s, dev = 2.7s //tensorflow/python:control_flow_ops_test_cpu PASSED in 33.7s Stats over 2 runs: max = 33.7s, min = 26.2s, avg = 30.0s, dev = 3.8s //tensorflow/python/data/experimental/kernel_tests/optimization:optimization_test PASSED in 19.1s Stats over 2 runs: max = 19.1s, min = 12.4s, avg = 15.8s, dev = 3.3s //tensorflow/python/data/experimental/kernel_tests/service:metadata_test PASSED in 18.5s Stats over 2 runs: max = 18.5s, min = 17.8s, avg = 18.2s, dev = 0.4s //tensorflow/python/data/kernel_tests:padded_batch_test PASSED in 28.3s Stats over 2 runs: max = 28.3s, min = 27.9s, avg = 28.1s, dev = 0.2s //tensorflow/python/data/kernel_tests:repeat_test PASSED in 67.5s Stats over 2 runs: max = 67.5s, min = 62.5s, avg = 65.0s, dev = 2.5s //tensorflow/python/data/kernel_tests:window_test PASSED in 40.6s Stats over 2 runs: max = 40.6s, min = 30.4s, avg = 35.5s, dev = 5.1s //tensorflow/python/kernel_tests/array_ops:scatter_nd_ops_test_cpu PASSED in 16.0s Stats over 2 runs: max = 16.0s, min = 15.8s, avg = 15.9s, dev = 0.1s //tensorflow/python/kernel_tests/control_flow:functional_ops_test_cpu PASSED in 16.2s Stats over 2 runs: max = 16.2s, min = 15.8s, avg = 16.0s, dev = 0.2s //tensorflow/python/kernel_tests/control_flow:map_fn_test_cpu PASSED in 13.5s Stats over 2 runs: max = 13.5s, min = 10.8s, avg = 12.1s, dev = 1.3s //tensorflow/python/kernel_tests/nn_ops:atrous_conv2d_test_cpu PASSED in 37.8s Stats over 2 runs: max = 37.8s, min = 30.5s, avg = 34.2s, dev = 3.6s //tensorflow/python/kernel_tests/nn_ops:bias_op_d9m_test_cpu PASSED in 105.7s Stats over 2 runs: max = 105.7s, min = 45.8s, avg = 75.7s, dev = 29.9s //tensorflow/python/kernel_tests/nn_ops:conv2d_backprop_filter_grad_test_cpu PASSED in 176.4s Stats over 2 runs: max = 176.4s, min = 9.8s, avg = 93.1s, dev = 83.3s //tensorflow/compiler/tests:spacetobatch_op_test_cpu PASSED in 9.0s Stats over 3 runs: max = 9.0s, min = 8.8s, avg = 8.9s, dev = 0.1s //tensorflow/compiler/tests:spacetobatch_op_test_cpu_mlir_bridge_test PASSED in 11.1s Stats over 3 runs: max = 11.1s, min = 10.0s, avg = 10.6s, dev = 0.5s //tensorflow/compiler/xla/tests:triangular_solve_test_cpu PASSED in 66.8s Stats over 3 runs: max = 66.8s, min = 64.6s, avg = 66.0s, dev = 1.0s //tensorflow/core/data/service:thread_safe_buffer_test PASSED in 0.7s Stats over 3 runs: max = 0.7s, min = 0.2s, avg = 0.5s, dev = 0.2s //tensorflow/python/data/experimental/kernel_tests/service:multi_process_cluster_test PASSED in 15.2s Stats over 3 runs: max = 15.2s, min = 10.9s, avg = 13.6s, dev = 1.9s //tensorflow/python/data/kernel_tests:unique_test PASSED in 14.3s Stats over 3 runs: max = 14.3s, min = 11.2s, avg = 12.2s, dev = 1.4s //tensorflow/python/kernel_tests/array_ops:gather_op_test_cpu PASSED in 58.3s Stats over 3 runs: max = 58.3s, min = 38.0s, avg = 46.1s, dev = 8.8s //tensorflow/python/kernel_tests/array_ops:weights_broadcast_test PASSED in 8.8s Stats over 3 runs: max = 8.8s, min = 8.1s, avg = 8.4s, dev = 0.3s //tensorflow/python/kernel_tests/distributions:util_test_cpu PASSED in 19.4s Stats over 3 runs: max = 19.4s, min = 11.9s, avg = 15.2s, dev = 3.1s //tensorflow/python/kernel_tests/linalg:matrix_triangular_solve_op_test_cpu PASSED in 661.0s Stats over 3 runs: max = 661.0s, min = 8.8s, avg = 226.5s, dev = 307.2s //tensorflow/python/kernel_tests/random:multinomial_op_big_test_cpu PASSED in 18.8s Stats over 3 runs: max = 18.8s, min = 11.6s, avg = 14.5s, dev = 3.1s //tensorflow/compiler/xla/tests:dynamic_ops_test_cpu PASSED in 22.3s Stats over 4 runs: max = 22.3s, min = 16.2s, avg = 18.6s, dev = 2.3s //tensorflow/core/kernels:example_parsing_ops_test PASSED in 0.6s Stats over 4 runs: max = 0.6s, min = 0.5s, avg = 0.6s, dev = 0.1s //tensorflow/python:nn_batchnorm_test_cpu PASSED in 30.6s Stats over 4 runs: max = 30.6s, min = 13.2s, avg = 17.8s, dev = 7.4s //tensorflow/python:nn_fused_batchnorm_d9m_test_cpu PASSED in 17.0s Stats over 4 runs: max = 17.0s, min = 14.9s, avg = 16.0s, dev = 0.8s //tensorflow/python/data/experimental/kernel_tests:auto_shard_dataset_test PASSED in 49.6s Stats over 4 runs: max = 49.6s, min = 25.1s, avg = 38.6s, dev = 9.7s //tensorflow/python/data/experimental/kernel_tests:map_and_batch_test PASSED in 41.3s Stats over 4 runs: max = 41.3s, min = 25.0s, avg = 30.0s, dev = 6.6s //tensorflow/python/data/experimental/kernel_tests:parse_example_dataset_test PASSED in 41.6s Stats over 4 runs: max = 41.6s, min = 15.3s, avg = 27.9s, dev = 11.9s //tensorflow/python/data/experimental/kernel_tests:rebatch_dataset_test PASSED in 31.1s Stats over 4 runs: max = 31.1s, min = 12.4s, avg = 19.7s, dev = 6.9s //tensorflow/python/data/experimental/kernel_tests:sql_dataset_test PASSED in 83.9s Stats over 4 runs: max = 83.9s, min = 55.9s, avg = 70.6s, dev = 13.2s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_ft_test PASSED in 11.0s Stats over 4 runs: max = 11.0s, min = 6.3s, avg = 8.7s, dev = 1.8s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_test PASSED in 46.7s Stats over 4 runs: max = 46.7s, min = 22.8s, avg = 33.9s, dev = 10.9s //tensorflow/python/data/kernel_tests:batch_test PASSED in 47.0s Stats over 4 runs: max = 47.0s, min = 39.2s, avg = 43.1s, dev = 2.8s //tensorflow/python/data/kernel_tests:fixed_length_record_dataset_test PASSED in 17.4s Stats over 4 runs: max = 17.4s, min = 10.1s, avg = 13.8s, dev = 3.5s //tensorflow/python/data/kernel_tests:from_generator_test PASSED in 27.5s Stats over 4 runs: max = 27.5s, min = 15.9s, avg = 20.7s, dev = 4.3s //tensorflow/python/data/kernel_tests:group_by_window_test PASSED in 32.4s Stats over 4 runs: max = 32.4s, min = 7.1s, avg = 16.8s, dev = 10.5s //tensorflow/python/data/kernel_tests:ragged_batch_test PASSED in 29.7s Stats over 4 runs: max = 29.7s, min = 25.9s, avg = 27.7s, dev = 1.4s //tensorflow/python/data/kernel_tests:skip_test PASSED in 38.3s Stats over 4 runs: max = 38.3s, min = 24.6s, avg = 31.1s, dev = 5.6s //tensorflow/python/data/kernel_tests:take_test PASSED in 25.4s Stats over 4 runs: max = 25.4s, min = 22.8s, avg = 24.2s, dev = 1.2s //tensorflow/python/data/kernel_tests:take_while_test PASSED in 30.8s Stats over 4 runs: max = 30.8s, min = 27.2s, avg = 28.6s, dev = 1.4s //tensorflow/python/data/kernel_tests:text_line_dataset_test PASSED in 21.0s Stats over 4 runs: max = 21.0s, min = 15.9s, avg = 18.4s, dev = 2.3s //tensorflow/python/data/kernel_tests:zip_test PASSED in 14.2s Stats over 4 runs: max = 14.2s, min = 12.2s, avg = 13.0s, dev = 0.7s //tensorflow/python/debug/lib:dumping_callback_test_cpu PASSED in 15.6s Stats over 4 runs: max = 15.6s, min = 13.6s, avg = 14.3s, dev = 0.8s //tensorflow/python/distribute:cross_device_ops_test_2gpu PASSED in 32.8s Stats over 4 runs: max = 32.8s, min = 25.1s, avg = 28.0s, dev = 3.1s //tensorflow/python/distribute:cross_device_ops_test_cpu PASSED in 30.5s Stats over 4 runs: max = 30.5s, min = 19.4s, avg = 25.0s, dev = 4.3s //tensorflow/python/framework:convert_to_constants_test PASSED in 38.1s Stats over 4 runs: max = 38.1s, min = 28.0s, avg = 32.4s, dev = 3.9s //tensorflow/python/kernel_tests:collective_ops_test_2gpu PASSED in 42.0s Stats over 4 runs: max = 42.0s, min = 40.8s, avg = 41.3s, dev = 0.4s //tensorflow/python/kernel_tests:collective_ops_test_cpu PASSED in 34.7s Stats over 4 runs: max = 34.7s, min = 33.1s, avg = 33.9s, dev = 0.7s //tensorflow/python/kernel_tests/array_ops:concat_op_test_cpu PASSED in 15.6s Stats over 4 runs: max = 15.6s, min = 12.0s, avg = 14.0s, dev = 1.3s //tensorflow/python/kernel_tests/array_ops:init_ops_test_cpu PASSED in 91.5s Stats over 4 runs: max = 91.5s, min = 30.1s, avg = 56.3s, dev = 25.0s //tensorflow/python/kernel_tests/array_ops:split_op_test_cpu PASSED in 33.8s Stats over 4 runs: max = 33.8s, min = 9.6s, avg = 21.6s, dev = 9.5s //tensorflow/python/kernel_tests/linalg:einsum_op_test_cpu PASSED in 96.9s Stats over 4 runs: max = 96.9s, min = 17.1s, avg = 50.6s, dev = 33.1s //tensorflow/python/kernel_tests/linalg:linear_operator_lower_triangular_test_cpu PASSED in 26.3s Stats over 4 runs: max = 26.3s, min = 24.5s, avg = 25.7s, dev = 0.7s //tensorflow/python/kernel_tests/random:random_gamma_test_cpu PASSED in 90.9s Stats over 4 runs: max = 90.9s, min = 7.3s, avg = 43.6s, dev = 36.9s //tensorflow/python/kernel_tests/signal:window_ops_test_cpu PASSED in 36.8s Stats over 4 runs: max = 36.8s, min = 35.4s, avg = 36.1s, dev = 0.5s //tensorflow/python/ops/ragged:ragged_gather_op_test PASSED in 69.0s Stats over 4 runs: max = 69.0s, min = 17.3s, avg = 40.3s, dev = 18.7s //tensorflow/python/ops/ragged:ragged_getitem_test PASSED in 70.4s Stats over 4 runs: max = 70.4s, min = 56.8s, avg = 62.8s, dev = 5.5s //tensorflow/compiler/tests:async_comp_test_cpu PASSED in 9.4s Stats over 5 runs: max = 9.4s, min = 8.8s, avg = 9.1s, dev = 0.2s //tensorflow/compiler/tests:conv3d_test_cpu PASSED in 13.9s Stats over 5 runs: max = 13.9s, min = 6.0s, avg = 9.3s, dev = 3.4s //tensorflow/compiler/tests:conv3d_test_cpu_mlir_bridge_test PASSED in 13.8s Stats over 5 runs: max = 13.8s, min = 6.7s, avg = 9.7s, dev = 3.0s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu PASSED in 13.0s Stats over 5 runs: max = 13.0s, min = 7.8s, avg = 9.7s, dev = 2.1s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu_mlir_bridge_test PASSED in 20.5s Stats over 5 runs: max = 20.5s, min = 11.0s, avg = 14.5s, dev = 3.8s //tensorflow/compiler/tests:fused_batchnorm_test_cpu PASSED in 11.2s Stats over 5 runs: max = 11.2s, min = 7.5s, avg = 9.0s, dev = 1.2s //tensorflow/compiler/tests:fused_batchnorm_test_cpu_mlir_bridge_test PASSED in 9.2s Stats over 5 runs: max = 9.2s, min = 7.9s, avg = 8.5s, dev = 0.4s //tensorflow/compiler/tests:image_ops_jit_compile_test_cpu PASSED in 8.0s Stats over 5 runs: max = 8.0s, min = 7.2s, avg = 7.5s, dev = 0.2s //tensorflow/compiler/tests:reduce_ops_test_cpu PASSED in 11.2s Stats over 5 runs: max = 11.2s, min = 9.6s, avg = 10.3s, dev = 0.5s //tensorflow/compiler/tests:reduce_ops_test_cpu_mlir_bridge_test PASSED in 12.0s Stats over 5 runs: max = 12.0s, min = 10.6s, avg = 11.2s, dev = 0.7s //tensorflow/compiler/tests:repeat_op_test_cpu PASSED in 8.2s Stats over 5 runs: max = 8.2s, min = 5.1s, avg = 6.2s, dev = 1.2s //tensorflow/compiler/tests:repeat_op_test_cpu_mlir_bridge_test PASSED in 9.1s Stats over 5 runs: max = 9.1s, min = 6.9s, avg = 7.6s, dev = 0.8s //tensorflow/compiler/tests:special_math_test_cpu PASSED in 130.4s Stats over 5 runs: max = 130.4s, min = 14.4s, avg = 53.5s, dev = 40.5s //tensorflow/compiler/tests:special_math_test_cpu_mlir_bridge_test PASSED in 102.5s Stats over 5 runs: max = 102.5s, min = 14.5s, avg = 48.8s, dev = 30.1s //tensorflow/compiler/xla/client/lib:self_adjoint_eig_test_cpu PASSED in 32.2s Stats over 5 runs: max = 32.2s, min = 15.6s, avg = 25.2s, dev = 6.6s //tensorflow/core/grappler/optimizers:constant_folding_test PASSED in 6.9s Stats over 5 runs: max = 6.9s, min = 4.4s, avg = 5.3s, dev = 1.0s //tensorflow/dtensor/python/tests:layout_propagation_test_cpu PASSED in 14.2s Stats over 5 runs: max = 14.2s, min = 8.4s, avg = 10.2s, dev = 2.1s //tensorflow/python/distribute:mirrored_strategy_test_2gpu PASSED in 20.4s Stats over 5 runs: max = 20.4s, min = 16.2s, avg = 18.2s, dev = 1.5s //tensorflow/python/distribute:mirrored_strategy_test_cpu PASSED in 13.2s Stats over 5 runs: max = 13.2s, min = 11.6s, avg = 12.5s, dev = 0.6s //tensorflow/python/distribute:moving_averages_test_2gpu PASSED in 19.9s Stats over 5 runs: max = 19.9s, min = 13.4s, avg = 16.5s, dev = 2.1s //tensorflow/python/distribute:moving_averages_test_cpu PASSED in 18.4s Stats over 5 runs: max = 18.4s, min = 11.8s, avg = 15.8s, dev = 2.4s //tensorflow/python/distribute:vars_test_2gpu PASSED in 24.5s Stats over 5 runs: max = 24.5s, min = 16.8s, avg = 20.2s, dev = 2.7s //tensorflow/python/distribute:vars_test_cpu PASSED in 19.1s Stats over 5 runs: max = 19.1s, min = 17.7s, avg = 18.4s, dev = 0.5s //tensorflow/python/eager:device_placement_test_cpu PASSED in 9.0s Stats over 5 runs: max = 9.0s, min = 6.5s, avg = 8.1s, dev = 0.8s //tensorflow/python/eager:forwardprop_test_cpu PASSED in 174.9s Stats over 5 runs: max = 174.9s, min = 20.4s, avg = 67.3s, dev = 55.3s //tensorflow/python/eager/polymorphic_function:gradients_test_cpu PASSED in 17.8s Stats over 5 runs: max = 17.8s, min = 11.8s, avg = 14.9s, dev = 2.5s //tensorflow/python/kernel_tests/linalg:cholesky_op_test_cpu PASSED in 88.2s Stats over 5 runs: max = 88.2s, min = 37.3s, avg = 52.1s, dev = 18.6s //tensorflow/python/kernel_tests/linalg:linear_operator_adjoint_test_cpu PASSED in 32.2s Stats over 5 runs: max = 32.2s, min = 26.2s, avg = 29.1s, dev = 2.4s //tensorflow/python/kernel_tests/linalg:linear_operator_composition_test_cpu PASSED in 41.7s Stats over 5 runs: max = 41.7s, min = 37.4s, avg = 39.7s, dev = 1.8s //tensorflow/python/kernel_tests/linalg:linear_operator_diag_test_cpu PASSED in 28.3s Stats over 5 runs: max = 28.3s, min = 22.5s, avg = 26.0s, dev = 2.0s //tensorflow/python/kernel_tests/linalg:linear_operator_full_matrix_test_cpu PASSED in 33.2s Stats over 5 runs: max = 33.2s, min = 27.3s, avg = 29.6s, dev = 2.1s //tensorflow/python/kernel_tests/linalg:linear_operator_householder_test_cpu PASSED in 27.1s Stats over 5 runs: max = 27.1s, min = 23.1s, avg = 25.4s, dev = 1.3s //tensorflow/python/kernel_tests/linalg:linear_operator_identity_test_cpu PASSED in 53.1s Stats over 5 runs: max = 53.1s, min = 36.6s, avg = 45.0s, dev = 6.6s //tensorflow/python/kernel_tests/linalg:linear_operator_inversion_test_cpu PASSED in 26.6s Stats over 5 runs: max = 26.6s, min = 25.1s, avg = 26.1s, dev = 0.5s //tensorflow/python/kernel_tests/linalg:linear_operator_permutation_test_cpu PASSED in 24.5s Stats over 5 runs: max = 24.5s, min = 21.3s, avg = 23.6s, dev = 1.2s //tensorflow/python/kernel_tests/linalg:linear_operator_toeplitz_test_cpu PASSED in 21.4s Stats over 5 runs: max = 21.4s, min = 16.8s, avg = 19.9s, dev = 1.6s //tensorflow/python/kernel_tests/linalg:linear_operator_tridiag_test_cpu PASSED in 101.3s Stats over 5 runs: max = 101.3s, min = 93.0s, avg = 96.0s, dev = 3.4s //tensorflow/python/kernel_tests/linalg:linear_operator_util_test_cpu PASSED in 9.3s Stats over 5 runs: max = 9.3s, min = 8.5s, avg = 8.8s, dev = 0.4s //tensorflow/python/kernel_tests/linalg:linear_operator_zeros_test_cpu PASSED in 13.5s Stats over 5 runs: max = 13.5s, min = 10.7s, avg = 12.4s, dev = 1.0s //tensorflow/python/kernel_tests/nn_ops:fractional_avg_pool_op_test PASSED in 13.8s Stats over 5 runs: max = 13.8s, min = 7.5s, avg = 9.4s, dev = 2.4s //tensorflow/python/kernel_tests/nn_ops:fractional_max_pool_op_test PASSED in 18.5s Stats over 5 runs: max = 18.5s, min = 7.3s, avg = 10.2s, dev = 4.2s //tensorflow/python/kernel_tests/sparse_ops:sparse_ops_test_cpu PASSED in 38.2s Stats over 5 runs: max = 38.2s, min = 9.1s, avg = 16.9s, dev = 10.7s //tensorflow/python/ops/parallel_for:math_test_cpu PASSED in 77.2s Stats over 5 runs: max = 77.2s, min = 23.5s, avg = 46.2s, dev = 19.4s //tensorflow/compiler/tests:scan_ops_test_cpu PASSED in 18.2s Stats over 6 runs: max = 18.2s, min = 12.6s, avg = 15.1s, dev = 1.8s //tensorflow/compiler/tests:scan_ops_test_cpu_mlir_bridge_test PASSED in 21.0s Stats over 6 runs: max = 21.0s, min = 12.3s, avg = 17.0s, dev = 2.6s //tensorflow/python:accumulate_n_benchmark_cpu PASSED in 6.9s Stats over 6 runs: max = 6.9s, min = 5.0s, avg = 6.1s, dev = 0.6s //tensorflow/python/data/experimental/kernel_tests:make_batched_features_dataset_test PASSED in 32.7s Stats over 6 runs: max = 32.7s, min = 6.4s, avg = 17.8s, dev = 11.0s //tensorflow/python/kernel_tests/array_ops:diag_op_test_cpu PASSED in 74.3s Stats over 6 runs: max = 74.3s, min = 8.8s, avg = 22.1s, dev = 23.4s //tensorflow/python/kernel_tests/math_ops:reduction_ops_test_cpu PASSED in 52.5s Stats over 6 runs: max = 52.5s, min = 24.2s, avg = 36.3s, dev = 8.9s //tensorflow/python/kernel_tests/nn_ops:conv_ops_test_cpu FAILED in 3 out of 6 in 35.3s Stats over 6 runs: max = 35.3s, min = 22.8s, avg = 28.7s, dev = 4.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu/shard_3_of_4/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu/shard_3_of_4/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/kernel_tests/nn_ops/conv_ops_test_cpu/shard_3_of_4/test_attempts/attempt_2.log //tensorflow/python/distribute/experimental/rpc:rpc_ops_test PASSED in 12.8s Stats over 7 runs: max = 12.8s, min = 8.9s, avg = 10.1s, dev = 1.5s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu PASSED in 71.6s Stats over 8 runs: max = 71.6s, min = 8.0s, avg = 29.2s, dev = 22.1s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu_mlir_bridge_test PASSED in 105.8s Stats over 8 runs: max = 105.8s, min = 6.1s, avg = 38.0s, dev = 33.7s //tensorflow/dtensor/python/tests:input_util_test PASSED in 38.3s Stats over 8 runs: max = 38.3s, min = 17.5s, avg = 27.1s, dev = 6.2s //tensorflow/python/data/experimental/kernel_tests:csv_dataset_test PASSED in 33.0s Stats over 8 runs: max = 33.0s, min = 8.3s, avg = 17.0s, dev = 9.4s //tensorflow/python/data/experimental/kernel_tests:parallel_interleave_test PASSED in 43.5s Stats over 8 runs: max = 43.5s, min = 14.0s, avg = 26.0s, dev = 8.9s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_ft_test PASSED in 48.5s Stats over 8 runs: max = 48.5s, min = 6.2s, avg = 25.3s, dev = 18.0s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_test PASSED in 23.3s Stats over 8 runs: max = 23.3s, min = 7.7s, avg = 13.1s, dev = 6.6s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_test PASSED in 20.3s Stats over 8 runs: max = 20.3s, min = 6.5s, avg = 12.7s, dev = 5.8s //tensorflow/python/data/experimental/kernel_tests/service:fault_tolerance_test PASSED in 21.7s Stats over 8 runs: max = 21.7s, min = 8.2s, avg = 11.5s, dev = 4.2s //tensorflow/python/data/kernel_tests:filter_test PASSED in 24.7s Stats over 8 runs: max = 24.7s, min = 15.3s, avg = 19.2s, dev = 3.4s //tensorflow/python/data/kernel_tests:flat_map_test PASSED in 41.5s Stats over 8 runs: max = 41.5s, min = 25.2s, avg = 33.4s, dev = 5.1s //tensorflow/python/data/kernel_tests:shard_test PASSED in 23.9s Stats over 8 runs: max = 23.9s, min = 17.0s, avg = 20.4s, dev = 2.1s //tensorflow/python/data/kernel_tests:shuffle_test PASSED in 80.8s Stats over 8 runs: max = 80.8s, min = 37.3s, avg = 47.7s, dev = 13.0s //tensorflow/python/data/kernel_tests:tf_record_dataset_test PASSED in 38.3s Stats over 8 runs: max = 38.3s, min = 18.8s, avg = 28.7s, dev = 5.9s //tensorflow/python/distribute/failure_handling:failure_handler_test PASSED in 71.2s Stats over 8 runs: max = 71.2s, min = 22.4s, avg = 43.1s, dev = 15.3s //tensorflow/python/kernel_tests/linalg:linalg_ops_test_cpu PASSED in 49.2s Stats over 8 runs: max = 49.2s, min = 29.0s, avg = 40.5s, dev = 7.3s //tensorflow/python/kernel_tests/linalg:linear_operator_block_diag_test_cpu PASSED in 67.1s Stats over 8 runs: max = 67.1s, min = 47.9s, avg = 58.3s, dev = 7.9s //tensorflow/python/kernel_tests/linalg:linear_operator_block_lower_triangular_test_cpu PASSED in 58.8s Stats over 8 runs: max = 58.8s, min = 38.3s, avg = 47.7s, dev = 7.4s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_d9m_test_cpu PASSED in 78.3s Stats over 8 runs: max = 78.3s, min = 3.9s, avg = 16.3s, dev = 23.9s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_test_cpu PASSED in 7.9s Stats over 8 runs: max = 7.9s, min = 6.2s, avg = 6.9s, dev = 0.6s //tensorflow/python/kernel_tests/signal:fft_ops_test_cpu PASSED in 24.7s Stats over 8 runs: max = 24.7s, min = 8.7s, avg = 14.5s, dev = 5.1s //tensorflow/python/ops/ragged:dynamic_ragged_shape_test PASSED in 61.7s Stats over 8 runs: max = 61.7s, min = 31.9s, avg = 43.1s, dev = 10.4s //tensorflow/python/ops/ragged:ragged_tensor_test PASSED in 50.3s Stats over 8 runs: max = 50.3s, min = 30.0s, avg = 35.4s, dev = 5.9s //tensorflow/python/distribute/failure_handling:gce_failure_handler_test FLAKY, failed in 1 out of 9 in 120.2s Stats over 9 runs: max = 120.2s, min = 11.7s, avg = 49.8s, dev = 43.8s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/failure_handling/gce_failure_handler_test/shard_7_of_8/test_attempts/attempt_1.log //tensorflow/compiler/tests:bincount_op_test_cpu PASSED in 7.4s Stats over 10 runs: max = 7.4s, min = 4.8s, avg = 5.8s, dev = 1.1s //tensorflow/compiler/tests:conv2d_test_cpu PASSED in 8.6s Stats over 10 runs: max = 8.6s, min = 5.6s, avg = 7.2s, dev = 1.1s //tensorflow/compiler/tests:conv2d_test_cpu_mlir_bridge_test PASSED in 7.9s Stats over 10 runs: max = 7.9s, min = 4.6s, avg = 6.4s, dev = 1.2s //tensorflow/compiler/tests:random_ops_test_cpu PASSED in 22.9s Stats over 10 runs: max = 22.9s, min = 14.6s, avg = 18.1s, dev = 2.5s //tensorflow/compiler/tests:random_ops_test_cpu_mlir_bridge_test PASSED in 19.4s Stats over 10 runs: max = 19.4s, min = 9.6s, avg = 14.7s, dev = 3.3s //tensorflow/compiler/tests:stateless_random_ops_test_cpu PASSED in 85.0s Stats over 10 runs: max = 85.0s, min = 44.1s, avg = 63.8s, dev = 13.8s //tensorflow/compiler/tests:stateless_random_ops_test_cpu_mlir_bridge_test PASSED in 67.2s Stats over 10 runs: max = 67.2s, min = 38.6s, avg = 53.3s, dev = 9.5s //tensorflow/compiler/xla/client/lib:svd_test_cpu PASSED in 69.1s Stats over 10 runs: max = 69.1s, min = 9.5s, avg = 27.3s, dev = 22.2s //tensorflow/compiler/xla/client/lib:tridiagonal_test_cpu PASSED in 12.7s Stats over 10 runs: max = 12.7s, min = 10.2s, avg = 11.1s, dev = 0.7s //tensorflow/compiler/xla/service/cpu:cpu_runtime_test PASSED in 13.0s Stats over 10 runs: max = 13.0s, min = 1.4s, avg = 9.0s, dev = 3.9s //tensorflow/python:special_math_ops_test_cpu PASSED in 48.6s Stats over 10 runs: max = 48.6s, min = 9.5s, avg = 15.5s, dev = 11.3s //tensorflow/python/data/kernel_tests:rejection_resample_test PASSED in 16.3s Stats over 10 runs: max = 16.3s, min = 6.4s, avg = 10.9s, dev = 3.4s //tensorflow/python/distribute:input_lib_test_2gpu PASSED in 42.5s Stats over 10 runs: max = 42.5s, min = 23.8s, avg = 34.6s, dev = 6.4s //tensorflow/python/distribute:input_lib_test_cpu PASSED in 58.9s Stats over 10 runs: max = 58.9s, min = 23.4s, avg = 33.7s, dev = 9.5s //tensorflow/python/distribute:input_lib_type_spec_test_2gpu PASSED in 21.9s Stats over 10 runs: max = 21.9s, min = 8.9s, avg = 15.2s, dev = 4.8s //tensorflow/python/distribute:input_lib_type_spec_test_cpu PASSED in 30.8s Stats over 10 runs: max = 30.8s, min = 18.6s, avg = 24.5s, dev = 3.8s //tensorflow/python/framework:config_vgpu_test_2gpu PASSED in 7.0s Stats over 10 runs: max = 7.0s, min = 4.7s, avg = 5.9s, dev = 0.8s //tensorflow/python/framework:config_vgpu_test_cpu PASSED in 9.3s Stats over 10 runs: max = 9.3s, min = 6.3s, avg = 7.9s, dev = 1.1s //tensorflow/python/framework:function_test_cpu PASSED in 61.6s Stats over 10 runs: max = 61.6s, min = 5.5s, avg = 14.8s, dev = 16.3s //tensorflow/python/grappler:cluster_test_cpu PASSED in 11.1s Stats over 10 runs: max = 11.1s, min = 8.6s, avg = 9.7s, dev = 0.7s //tensorflow/python/kernel_tests/array_ops:array_ops_test_cpu PASSED in 22.3s Stats over 10 runs: max = 22.3s, min = 13.4s, avg = 17.1s, dev = 2.7s //tensorflow/python/kernel_tests/array_ops:inplace_ops_test_cpu PASSED in 8.3s Stats over 10 runs: max = 8.3s, min = 4.6s, avg = 6.6s, dev = 1.3s //tensorflow/python/kernel_tests/data_structures:tensor_array_ops_test_cpu PASSED in 12.6s Stats over 10 runs: max = 12.6s, min = 8.8s, avg = 10.7s, dev = 1.2s //tensorflow/python/kernel_tests/linalg:linear_operator_low_rank_update_test_cpu PASSED in 94.2s Stats over 10 runs: max = 94.2s, min = 82.2s, avg = 88.1s, dev = 3.5s //tensorflow/python/kernel_tests/linalg:tridiagonal_matmul_op_test_cpu PASSED in 93.4s Stats over 10 runs: max = 93.4s, min = 6.8s, avg = 16.8s, dev = 25.6s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_ops_test_cpu PASSED in 39.3s Stats over 10 runs: max = 39.3s, min = 10.3s, avg = 24.1s, dev = 8.8s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_test_cpu PASSED in 40.1s Stats over 10 runs: max = 40.1s, min = 16.4s, avg = 26.6s, dev = 9.2s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_test_cpu PASSED in 25.4s Stats over 10 runs: max = 25.4s, min = 5.7s, avg = 11.1s, dev = 6.5s //tensorflow/python/kernel_tests/nn_ops:rnn_test_cpu PASSED in 10.9s Stats over 10 runs: max = 10.9s, min = 8.0s, avg = 9.7s, dev = 0.8s //tensorflow/python/kernel_tests/random:random_index_shuffle_test PASSED in 9.0s Stats over 10 runs: max = 9.0s, min = 6.0s, avg = 7.5s, dev = 0.9s //tensorflow/python/kernel_tests/random:stateless_random_ops_test_cpu PASSED in 130.5s Stats over 10 runs: max = 130.5s, min = 21.8s, avg = 73.9s, dev = 50.1s //tensorflow/python/ops/ragged:ragged_tensor_supported_values_test PASSED in 24.8s Stats over 10 runs: max = 24.8s, min = 18.1s, avg = 21.7s, dev = 2.1s //tensorflow/python/saved_model:load_test_cpu PASSED in 57.3s Stats over 10 runs: max = 57.3s, min = 31.3s, avg = 37.4s, dev = 7.2s //tensorflow/compiler/tests:fft_test_cpu PASSED in 21.5s Stats over 12 runs: max = 21.5s, min = 5.8s, avg = 13.0s, dev = 6.0s //tensorflow/compiler/xla/service:triangular_solve_expander_test PASSED in 7.2s Stats over 12 runs: max = 7.2s, min = 3.7s, avg = 5.4s, dev = 0.8s //tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 23.4s Stats over 12 runs: max = 23.4s, min = 3.2s, avg = 11.5s, dev = 5.8s //tensorflow/python/data/kernel_tests:choose_from_datasets_test PASSED in 12.6s Stats over 12 runs: max = 12.6s, min = 6.4s, avg = 9.1s, dev = 1.8s //tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu PASSED in 11.8s Stats over 12 runs: max = 11.8s, min = 5.7s, avg = 9.1s, dev = 1.8s //tensorflow/python/distribute:multi_process_runner_test_2gpu PASSED in 227.7s Stats over 12 runs: max = 227.7s, min = 14.9s, avg = 53.1s, dev = 58.4s //tensorflow/python/distribute:multi_process_runner_test_cpu PASSED in 226.9s Stats over 12 runs: max = 226.9s, min = 13.8s, avg = 52.1s, dev = 58.9s //tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 17.0s Stats over 15 runs: max = 17.0s, min = 9.5s, avg = 13.3s, dev = 2.0s //tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 82.7s Stats over 15 runs: max = 82.7s, min = 66.2s, avg = 74.9s, dev = 4.4s //tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu PASSED in 52.7s Stats over 15 runs: max = 52.7s, min = 9.2s, avg = 15.0s, dev = 10.9s //tensorflow/compiler/tests:ftrl_test_cpu PASSED in 8.9s Stats over 16 runs: max = 8.9s, min = 6.9s, avg = 7.9s, dev = 0.7s //tensorflow/compiler/tests:ternary_ops_test_cpu PASSED in 15.4s Stats over 16 runs: max = 15.4s, min = 7.2s, avg = 9.1s, dev = 2.1s //tensorflow/compiler/tests:ternary_ops_test_cpu_mlir_bridge_test PASSED in 18.3s Stats over 16 runs: max = 18.3s, min = 4.4s, avg = 7.7s, dev = 3.3s //tensorflow/python:image_ops_test_cpu PASSED in 19.1s Stats over 16 runs: max = 19.1s, min = 8.5s, avg = 12.5s, dev = 2.8s //tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 17.6s Stats over 16 runs: max = 17.6s, min = 4.8s, avg = 10.6s, dev = 4.0s //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test PASSED in 21.7s Stats over 16 runs: max = 21.7s, min = 5.6s, avg = 13.6s, dev = 5.1s //tensorflow/python/data/kernel_tests:snapshot_test PASSED in 29.6s Stats over 16 runs: max = 29.6s, min = 12.4s, avg = 19.4s, dev = 4.6s //tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 30.5s Stats over 16 runs: max = 30.5s, min = 7.3s, avg = 10.6s, dev = 5.3s //tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test PASSED in 10.8s Stats over 16 runs: max = 10.8s, min = 6.1s, avg = 8.1s, dev = 1.3s //tensorflow/python/kernel_tests/signal:dct_ops_test_cpu PASSED in 14.7s Stats over 16 runs: max = 14.7s, min = 9.2s, avg = 11.6s, dev = 1.3s //tensorflow/python/ops/parallel_for:control_flow_ops_test_cpu PASSED in 75.4s Stats over 16 runs: max = 75.4s, min = 14.8s, avg = 25.8s, dev = 13.4s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test PASSED in 10.7s Stats over 17 runs: max = 10.7s, min = 3.9s, avg = 6.8s, dev = 1.8s //tensorflow/python/data/kernel_tests:map_test PASSED in 38.5s Stats over 19 runs: max = 38.5s, min = 11.6s, avg = 19.9s, dev = 6.7s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu PASSED in 8.0s Stats over 20 runs: max = 8.0s, min = 4.1s, avg = 6.1s, dev = 1.3s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test PASSED in 10.2s Stats over 20 runs: max = 10.2s, min = 6.1s, avg = 7.4s, dev = 0.8s //tensorflow/compiler/tests:pooling_ops_test_cpu PASSED in 16.4s Stats over 20 runs: max = 16.4s, min = 3.8s, avg = 6.9s, dev = 2.8s //tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test PASSED in 17.7s Stats over 20 runs: max = 17.7s, min = 3.1s, avg = 6.4s, dev = 3.2s //tensorflow/compiler/xla/tests:convolution_dimension_numbers_test_cpu PASSED in 13.7s Stats over 20 runs: max = 13.7s, min = 10.3s, avg = 11.8s, dev = 0.9s //tensorflow/compiler/xla/tests:dot_operation_single_threaded_runtime_test_cpu PASSED in 18.1s Stats over 20 runs: max = 18.1s, min = 14.6s, avg = 16.3s, dev = 1.0s //tensorflow/compiler/xla/tests:dot_operation_test_cpu PASSED in 19.2s Stats over 20 runs: max = 19.2s, min = 14.2s, avg = 16.7s, dev = 1.3s //tensorflow/compiler/xla/tests:prng_test_cpu PASSED in 12.6s Stats over 20 runs: max = 12.6s, min = 10.5s, avg = 11.2s, dev = 0.5s //tensorflow/compiler/xla/tests:reduce_window_test_cpu PASSED in 50.6s Stats over 20 runs: max = 50.6s, min = 11.9s, avg = 24.8s, dev = 11.6s //tensorflow/python/autograph/tests:loop_control_flow_test PASSED in 19.1s Stats over 20 runs: max = 19.1s, min = 13.2s, avg = 16.5s, dev = 1.8s //tensorflow/python/kernel_tests:metrics_test PASSED in 45.2s Stats over 20 runs: max = 45.2s, min = 9.7s, avg = 21.8s, dev = 10.6s //tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu PASSED in 7.9s Stats over 20 runs: max = 7.9s, min = 3.2s, avg = 4.8s, dev = 1.6s //tensorflow/python/kernel_tests/data_structures:barrier_ops_test PASSED in 18.1s Stats over 20 runs: max = 18.1s, min = 5.7s, avg = 9.8s, dev = 3.1s //tensorflow/python/kernel_tests/linalg:eig_op_test PASSED in 43.9s Stats over 20 runs: max = 43.9s, min = 3.6s, avg = 14.0s, dev = 13.9s //tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu PASSED in 169.0s Stats over 20 runs: max = 169.0s, min = 40.1s, avg = 85.4s, dev = 34.9s //tensorflow/python/kernel_tests/linalg:norm_op_test_cpu PASSED in 9.7s Stats over 20 runs: max = 9.7s, min = 5.3s, avg = 7.2s, dev = 1.3s //tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu PASSED in 16.7s Stats over 20 runs: max = 16.7s, min = 8.5s, avg = 12.0s, dev = 2.5s //tensorflow/python/kernel_tests/linalg:qr_op_test_cpu PASSED in 229.4s Stats over 20 runs: max = 229.4s, min = 32.1s, avg = 102.6s, dev = 55.2s //tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu PASSED in 23.0s Stats over 20 runs: max = 23.0s, min = 4.5s, avg = 11.6s, dev = 5.9s //tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu PASSED in 23.9s Stats over 20 runs: max = 23.9s, min = 6.2s, avg = 14.8s, dev = 5.8s //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu PASSED in 27.7s Stats over 20 runs: max = 27.7s, min = 16.8s, avg = 21.9s, dev = 2.6s //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu PASSED in 69.9s Stats over 20 runs: max = 69.9s, min = 9.6s, avg = 32.2s, dev = 19.6s //tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu PASSED in 20.0s Stats over 20 runs: max = 20.0s, min = 8.8s, avg = 14.4s, dev = 3.1s //tensorflow/python/data/experimental/kernel_tests/service:local_workers_test PASSED in 24.7s Stats over 24 runs: max = 24.7s, min = 6.9s, avg = 17.3s, dev = 4.4s //tensorflow/python/data/kernel_tests:interleave_test PASSED in 27.5s Stats over 24 runs: max = 27.5s, min = 10.1s, avg = 17.0s, dev = 4.9s //tensorflow/python/data/kernel_tests:sample_from_datasets_test PASSED in 24.0s Stats over 24 runs: max = 24.0s, min = 3.5s, avg = 10.5s, dev = 5.6s //tensorflow/compiler/xla/tests:array_elementwise_ops_test_cpu PASSED in 13.8s Stats over 25 runs: max = 13.8s, min = 11.2s, avg = 12.7s, dev = 0.8s //tensorflow/compiler/xla/tests:select_and_scatter_test_cpu PASSED in 48.4s Stats over 25 runs: max = 48.4s, min = 10.7s, avg = 18.8s, dev = 9.4s //tensorflow/compiler/xla/tests:convolution_variants_test_cpu PASSED in 13.5s Stats over 30 runs: max = 13.5s, min = 10.7s, avg = 12.2s, dev = 0.9s //tensorflow/compiler/xla/tests:iota_test_cpu PASSED in 26.2s Stats over 30 runs: max = 26.2s, min = 17.3s, avg = 20.0s, dev = 2.0s //tensorflow/compiler/xla/tests:params_test_cpu PASSED in 14.4s Stats over 30 runs: max = 14.4s, min = 10.7s, avg = 12.3s, dev = 0.9s //tensorflow/compiler/xla/tests:reshape_test_cpu PASSED in 14.1s Stats over 30 runs: max = 14.1s, min = 10.9s, avg = 12.7s, dev = 1.0s //tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu PASSED in 33.4s Stats over 30 runs: max = 33.4s, min = 3.3s, avg = 11.1s, dev = 7.4s //tensorflow/compiler/xla/tests:reduce_test_cpu PASSED in 15.8s Stats over 31 runs: max = 15.8s, min = 10.5s, avg = 13.5s, dev = 1.2s //tensorflow/compiler/tests:unary_ops_test_cpu PASSED in 19.7s Stats over 32 runs: max = 19.7s, min = 3.0s, avg = 6.3s, dev = 3.7s //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test PASSED in 30.4s Stats over 32 runs: max = 30.4s, min = 3.0s, avg = 7.8s, dev = 5.3s //tensorflow/compiler/xla/tests:scalar_computations_test_cpu PASSED in 19.2s Stats over 32 runs: max = 19.2s, min = 10.8s, avg = 13.8s, dev = 1.8s //tensorflow/python/data/experimental/kernel_tests/service:auto_shard_test PASSED in 19.2s Stats over 32 runs: max = 19.2s, min = 3.7s, avg = 13.0s, dev = 3.6s //tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 26.8s Stats over 32 runs: max = 26.8s, min = 3.2s, avg = 11.4s, dev = 5.3s //tensorflow/compiler/xla/tests:batch_normalization_test_cpu PASSED in 17.4s Stats over 40 runs: max = 17.4s, min = 11.8s, avg = 14.9s, dev = 1.3s //tensorflow/compiler/xla/tests:bfloat16_test_cpu PASSED in 18.4s Stats over 40 runs: max = 18.4s, min = 12.4s, avg = 15.1s, dev = 1.5s //tensorflow/compiler/xla/tests:conv_depthwise_backprop_filter_test_cpu PASSED in 16.4s Stats over 40 runs: max = 16.4s, min = 13.6s, avg = 15.1s, dev = 0.9s //tensorflow/compiler/xla/tests:slice_test_cpu PASSED in 20.8s Stats over 40 runs: max = 20.8s, min = 11.2s, avg = 15.0s, dev = 2.0s //tensorflow/core/kernels:stochastic_cast_op_test PASSED in 3.5s Stats over 48 runs: max = 3.5s, min = 0.7s, avg = 1.4s, dev = 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 52.1s Stats over 50 runs: max = 52.1s, min = 17.0s, avg = 29.8s, dev = 8.5s //tensorflow/compiler/tests:sort_ops_test_cpu PASSED in 48.0s Stats over 50 runs: max = 48.0s, min = 2.8s, avg = 12.5s, dev = 9.4s //tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test PASSED in 44.3s Stats over 50 runs: max = 44.3s, min = 2.7s, avg = 10.7s, dev = 8.2s //tensorflow/compiler/xla/tests:conv_depthwise_test_cpu PASSED in 16.3s Stats over 50 runs: max = 16.3s, min = 12.9s, avg = 14.9s, dev = 0.9s //tensorflow/compiler/xla/tests:convolution_test_1d_no_vmodule_cpu PASSED in 21.6s Stats over 50 runs: max = 21.6s, min = 17.1s, avg = 19.4s, dev = 1.2s //tensorflow/compiler/xla/tests:convolution_test_cpu PASSED in 22.2s Stats over 50 runs: max = 22.2s, min = 13.9s, avg = 18.0s, dev = 1.7s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 10.6s Stats over 50 runs: max = 10.6s, min = 4.3s, avg = 7.3s, dev = 1.8s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_grad_test_cpu PASSED in 8.8s Stats over 50 runs: max = 8.8s, min = 2.9s, avg = 4.3s, dev = 1.3s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_sparse_mat_mul_grad_test_cpu PASSED in 7.5s Stats over 50 runs: max = 7.5s, min = 3.1s, avg = 4.3s, dev = 1.0s //tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu PASSED in 36.9s Stats over 50 runs: max = 36.9s, min = 8.5s, avg = 17.8s, dev = 7.0s //tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu PASSED in 13.9s Stats over 50 runs: max = 13.9s, min = 3.7s, avg = 5.9s, dev = 1.8s //tensorflow/python/kernel_tests/math_ops:cwise_ops_unary_test_cpu PASSED in 14.4s Stats over 50 runs: max = 14.4s, min = 2.8s, avg = 5.8s, dev = 2.6s Executed 3747 out of 3747 tests: 3746 tests pass and 1 fails locally. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.