==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memcpy.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_memcpy.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memcpy.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_memcpy.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memcpy.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_memcpy.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/xla_cpu/tests:ops.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/xla_cpu/tests/ops.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/xla_cpu/tests:ops.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/xla_cpu/tests/ops.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/xla_cpu/tests:ops.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/xla_cpu/tests/ops.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_outfeed.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_outfeed.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_outfeed.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_outfeed.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_outfeed.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_outfeed.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:memref_get_global_to_arg.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/memref_get_global_to_arg.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:memref_get_global_to_arg.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/memref_get_global_to_arg.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:memref_get_global_to_arg.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/memref_get_global_to_arg.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_large_constants.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_large_constants.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_large_constants.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_large_constants.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_large_constants.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_large_constants.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import_async.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/import_async.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import_async.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/import_async.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import_async.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/import_async.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:return-operands-of-terminator-operands.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/return-operands-of-terminator-operands.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:return-operands-of-terminator-operands.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/return-operands-of-terminator-operands.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:return-operands-of-terminator-operands.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/return-operands-of-terminator-operands.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cholesky.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_cholesky.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cholesky.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_cholesky.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cholesky.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_cholesky.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:outline_cuda_graphs.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/outline_cuda_graphs.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:outline_cuda_graphs.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/outline_cuda_graphs.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:outline_cuda_graphs.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/outline_cuda_graphs.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_infeed.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_infeed.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_infeed.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_infeed.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_infeed.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_infeed.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:layouts_and_names.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/layouts_and_names.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:layouts_and_names.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/layouts_and_names.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:layouts_and_names.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/layouts_and_names.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:rng_get_and_update_state.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/rng_get_and_update_state.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:rng_get_and_update_state.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/rng_get_and_update_state.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:rng_get_and_update_state.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/rng_get_and_update_state.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:if.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/if.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:if.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/if.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:if.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/if.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:spmd_module_sharding.hlo.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/spmd_module_sharding.hlo.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:spmd_module_sharding.hlo.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/spmd_module_sharding.hlo.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:spmd_module_sharding.hlo.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/spmd_module_sharding.hlo.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_stacktrace.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/location_to_stacktrace.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_stacktrace.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/location_to_stacktrace.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_stacktrace.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/location_to_stacktrace.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:frontend_attributes.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/frontend_attributes.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:frontend_attributes.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/frontend_attributes.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:frontend_attributes.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/frontend_attributes.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:lmhlo_custom_call.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/lmhlo_custom_call.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:lmhlo_custom_call.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/lmhlo_custom_call.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:lmhlo_custom_call.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/lmhlo_custom_call.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:simple.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/simple.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:simple.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/simple.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:simple.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/simple.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:unsupported_type.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/unsupported_type.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:unsupported_type.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/unsupported_type.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:unsupported_type.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/unsupported_type.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:missing_main.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/missing_main.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:missing_main.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/missing_main.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:missing_main.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/missing_main.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_infeed.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_infeed.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_infeed.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_infeed.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_infeed.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_infeed.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:while.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/while.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:while.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/while.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:while.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/while.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_assemble.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_assemble.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_assemble.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_assemble.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_assemble.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_assemble.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:opaque_elements_attr.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/opaque_elements_attr.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:opaque_elements_attr.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/opaque_elements_attr.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:opaque_elements_attr.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/opaque_elements_attr.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_reshard.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_reshard.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_reshard.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_reshard.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_reshard.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_reshard.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:fusion.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/fusion.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:fusion.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/fusion.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:fusion.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/fusion.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:stacktrace_to_location.hlo.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/stacktrace_to_location.hlo.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:stacktrace_to_location.hlo.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/stacktrace_to_location.hlo.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:stacktrace_to_location.hlo.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/stacktrace_to_location.hlo.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:while.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/while.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:while.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/while.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:while.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/while.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/xla_cpu/tests:bufferize.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/xla_cpu/tests/bufferize.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/xla_cpu/tests:bufferize.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/xla_cpu/tests/bufferize.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/xla_cpu/tests:bufferize.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/xla_cpu/tests/bufferize.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/math/transforms/tests:math_optimization.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/math/transforms/tests/math_optimization.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/math/transforms/tests:math_optimization.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/math/transforms/tests/math_optimization.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/math/transforms/tests:math_optimization.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/math/transforms/tests/math_optimization.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:case.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/case.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:case.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/case.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:case.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/case.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/python/distribute/failure_handling:failure_handler_test (shard 5 of 8): Running tests under Python 3.11.1: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/python_aarch64-unknown-linux-gnu/bin/python3 [ RUN ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice INFO:tensorflow:Start watcher for local signal. I0725 23:19:28.059360 281472975207104 failure_handling.py:674] Start watcher for local signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. I0725 23:19:28.059739 281472975207104 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. W0725 23:19:28.060069 281472975207104 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. INFO:tensorflow:Start training at 0 I0725 23:19:28.060293 281472975207104 failure_handler_test.py:197] Start training at 0 WARNING:tensorflow:5 out of the last 5 calls to .distributed_train_step..train_step at 0xfffedbca27a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0725 23:19:28.384599 281472975207104 polymorphic_function.py:156] 5 out of the last 5 calls to .distributed_train_step..train_step at 0xfffedbca27a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. WARNING:tensorflow:6 out of the last 6 calls to .distributed_train_step..train_step at 0xfffedbca27a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0725 23:19:28.399386 281472975207104 polymorphic_function.py:156] 6 out of the last 6 calls to .distributed_train_step..train_step at 0xfffedbca27a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. INFO:tensorflow:epoch 0 finished I0725 23:19:29.032489 281472975207104 failure_handler_test.py:195] epoch 0 finished INFO:tensorflow:epoch 1 finished I0725 23:19:29.307613 281472975207104 failure_handler_test.py:195] epoch 1 finished INFO:tensorflow:epoch 2 finished I0725 23:19:29.611950 281472975207104 failure_handler_test.py:195] epoch 2 finished INFO:tensorflow:epoch 3 finished I0725 23:19:30.151004 281472975207104 failure_handler_test.py:195] epoch 3 finished INFO:tensorflow:epoch 4 finished I0725 23:19:30.380516 281472975207104 failure_handler_test.py:195] epoch 4 finished INFO:tensorflow:epoch 5 finished I0725 23:19:30.813433 281472975207104 failure_handler_test.py:195] epoch 5 finished INFO:tensorflow:epoch 6 finished I0725 23:19:31.027802 281472975207104 failure_handler_test.py:195] epoch 6 finished INFO:tensorflow:epoch 7 finished I0725 23:19:31.241832 281472975207104 failure_handler_test.py:195] epoch 7 finished INFO:tensorflow:Training finished. I0725 23:19:31.242392 281472975207104 failure_handler_test.py:245] Training finished. INFO:tensorflow:sending sigterm I0725 23:19:31.566313 281470072189408 failure_handler_test.py:467] sending sigterm INFO:tensorflow:Member single_worker has received termination notice. I0725 23:19:31.566817 281472975207104 failure_handling.py:701] Member single_worker has received termination notice. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice): 4.05s I0725 23:19:31.567345 281472975207104 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice): 4.05s [ OK ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 38063 I0725 23:19:31.571149 281472975207104 test_util.py:3813] Using local port 38063 INFO:tensorflow:Using local port 42713 I0725 23:19:31.571552 281472975207104 test_util.py:3813] Using local port 42713 INFO:tensorflow:Using local port 38057 I0725 23:19:31.571907 281472975207104 test_util.py:3813] Using local port 38057 INFO:tensorflow:Using local port 34867 I0725 23:19:31.572250 281472975207104 test_util.py:3813] Using local port 34867 INFO:tensorflow:Cluster starting. I0725 23:19:36.151499 281472975207104 failure_handler_test.py:297] Cluster starting. [worker-1]: I0725 23:19:36.395831 281472972651200 multi_process_runner.py:840] Subprocess with PID 929279 (worker, 1) is now being started. [worker-0]: I0725 23:19:36.425601 281472972651200 multi_process_runner.py:840] Subprocess with PID 929266 (worker, 0) is now being started. [worker-1]: I0725 23:19:36.396270 281472972651200 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38063", "localhost:42713", "localhost:38057", "localhost:34867"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: I0725 23:19:36.426033 281472972651200 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38063", "localhost:42713", "localhost:38057", "localhost:34867"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: I0725 23:19:36.451272 281472972651200 multi_process_runner.py:840] Subprocess with PID 929388 (worker, 2) is now being started. [worker-2]: I0725 23:19:36.451669 281472972651200 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38063", "localhost:42713", "localhost:38057", "localhost:34867"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-2]: 2023-07-25 23:19:36.485933: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:38057 [worker-3]: I0725 23:19:36.487948 281472972651200 multi_process_runner.py:840] Subprocess with PID 929394 (worker, 3) is now being started. [worker-3]: I0725 23:19:36.488356 281472972651200 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38063", "localhost:42713", "localhost:38057", "localhost:34867"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2023-07-25 23:19:36.552136: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:38063 [worker-3]: 2023-07-25 23:19:36.553141: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:34867 [worker-1]: 2023-07-25 23:19:36.563242: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:42713 [worker-0]: 2023-07-25 23:19:36.587512: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 3757081286627078594 [worker-0]: 2023-07-25 23:19:36.587765: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 17115562008170436944 [worker-0]: 2023-07-25 23:19:36.588273: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-07-25 23:19:36.592565: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 15173180263086164160 [worker-1]: 2023-07-25 23:19:36.596436: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-2]: 2023-07-25 23:19:36.599166: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-07-25 23:19:36.616430: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 7270171761884902993 [worker-3]: 2023-07-25 23:19:36.617547: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0725 23:19:36.619681 281472972651200 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0725 23:19:36.620901 281472972651200 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0725 23:19:36.619908 281472972651200 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0725 23:19:36.637275 281472972651200 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0725 23:19:36.675280 281472972651200 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0725 23:19:36.675885 281472972651200 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38063', 'localhost:42713', 'localhost:38057', 'localhost:34867']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0725 23:19:36.676143 281472972651200 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38063', 'localhost:42713', 'localhost:38057', 'localhost:34867']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0725 23:19:36.676069 281472972651200 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0725 23:19:36.676571 281472972651200 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38063', 'localhost:42713', 'localhost:38057', 'localhost:34867']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0725 23:19:36.676799 281472972651200 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38063', 'localhost:42713', 'localhost:38057', 'localhost:34867']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0725 23:19:36.692106 281472972651200 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0725 23:19:36.692654 281472972651200 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38063', 'localhost:42713', 'localhost:38057', 'localhost:34867']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0725 23:19:36.692877 281472972651200 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38063', 'localhost:42713', 'localhost:38057', 'localhost:34867']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0725 23:19:36.801522 281472972651200 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0725 23:19:36.802787 281472972651200 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38063', 'localhost:42713', 'localhost:38057', 'localhost:34867']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0725 23:19:36.803018 281472972651200 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38063', 'localhost:42713', 'localhost:38057', 'localhost:34867']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0725 23:19:37.016394 281472972651200 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0725 23:19:37.017524 281472972651200 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: I0725 23:19:37.007529 281472972651200 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I0725 23:19:37.026901 281472972651200 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I0725 23:19:37.027641 281472972651200 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0725 23:19:37.027992 281472972651200 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: I0725 23:19:37.027238 281472972651200 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-2]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0725 23:19:37.028314 281472972651200 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: W0725 23:19:37.027579 281472972651200 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-2]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-2]: INFO:tensorflow:Start training at 0 [worker-0]: I0725 23:19:37.028502 281472972651200 failure_handler_test.py:197] Start training at 0 [worker-2]: I0725 23:19:37.027779 281472972651200 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0725 23:19:37.036755 281472972651200 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0725 23:19:37.037077 281472972651200 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0725 23:19:37.037399 281472972651200 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0725 23:19:37.037585 281472972651200 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0725 23:19:37.038166 281472972651200 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I0725 23:19:37.038927 281472972651200 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0725 23:19:37.039154 281472972651200 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0725 23:19:37.039454 281472972651200 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0725 23:19:37.039635 281472972651200 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:37.224430 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:37.263327 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:37.266866 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:37.399004 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:37.533396 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:37.534456 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:37.548741 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:37.545727 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:37.625874 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:37.635947 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:37.628843 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:37.650907 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:37.707555 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:37.715187 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:37.707713 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:37.726913 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:37.796250 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:37.820518 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:37.824016 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:37.840600 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff84202a20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff84200c20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0725 23:19:37.891680 281472972651200 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff84202a20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff842027a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0725 23:19:37.897197 281472972651200 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff842027a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff84200d60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0725 23:19:37.897653 281472972651200 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff84200d60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0725 23:19:37.891564 281472972651200 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff84200c20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:37.900683 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:37.914001 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:37.922425 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:37.951442 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff842009a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0725 23:19:38.108726 281472972651200 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff842009a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff842032e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff84203880> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0725 23:19:38.116367 281472972651200 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff84203880> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff84201f80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0725 23:19:38.114740 281472972651200 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff84201f80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0725 23:19:38.115198 281472972651200 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff842032e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:38.125077 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:38.127611 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:38.121272 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:38.148082 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:38.288141 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:38.303523 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:38.310784 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:38.321181 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:38.417859 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:38.435365 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:38.447536 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:38.463410 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:38.575267 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:38.575268 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:38.584407 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:38.600310 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:38.677079 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:38.679487 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:38.680629 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:38.680217 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:39.117261 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:39.123539 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:39.130362 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:39.125005 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:39.237810 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:39.238641 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:39.246814 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:39.260415 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:39.360381 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:39.377258 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:39.389832 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:39.392890 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:39.447948 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:39.455612 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:39.458889 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:39.447952 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 0 finished [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0725 23:19:39.505486 281472972651200 failure_handler_test.py:195] epoch 0 finished [worker-3]: I0725 23:19:39.505395 281472972651200 failure_handler_test.py:195] epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0725 23:19:39.506438 281472972651200 failure_handler_test.py:195] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:39.514986 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:39.515014 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0725 23:19:39.516585 281472972651200 failure_handler_test.py:195] epoch 0 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:39.519774 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:39.525636 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:39.583774 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:39.583765 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:39.586311 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:39.600429 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:39.715812 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:39.717571 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:39.719433 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:39.724600 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:39.786822 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:39.787122 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:39.786382 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:39.807039 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:39.904878 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:39.905022 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:39.905536 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:39.910544 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:39.985981 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:39.985977 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:40.000353 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:40.019787 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:40.109879 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:40.135147 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:40.140315 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:40.140299 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:40.346002 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:40.349879 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:40.380346 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:40.371689 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending sigterm I0725 23:19:40.432277 281472975207104 failure_handler_test.py:302] sending sigterm INFO:tensorflow:sigterm sent I0725 23:19:40.432703 281472975207104 failure_handler_test.py:306] sigterm sent [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:40.472375 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Member 2 has received termination notice. [worker-2]: I0725 23:19:40.490465 281472972651200 failure_handling.py:710] Member 2 has received termination notice. [worker-2]: INFO:tensorflow:Termination caught in main thread on preempted worker [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-2]: I0725 23:19:40.491014 281472972651200 failure_handling.py:1159] Termination caught in main thread on preempted worker [worker-0]: I0725 23:19:40.493429 281446505640416 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-2]: I0725 23:19:40.493438 281446715421152 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:RUN_TO_CHECKPOINT set to 24 [worker-2]: I0725 23:19:40.494325 281472972651200 failure_handling.py:1168] RUN_TO_CHECKPOINT set to 24 [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 0 received [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:40.501655 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:40.499016 281472972651200 failure_handling.py:1177] Sigterm acknowledgement from replica 0 received [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-1]: I0725 23:19:40.506184 281446849638880 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:40.508865 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:40.506592 281447486910944 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 1 received [worker-2]: I0725 23:19:40.505960 281472972651200 failure_handling.py:1177] Sigterm acknowledgement from replica 1 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 2 received [worker-2]: I0725 23:19:40.506773 281472972651200 failure_handling.py:1177] Sigterm acknowledgement from replica 2 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 3 received [worker-2]: I0725 23:19:40.507350 281472972651200 failure_handling.py:1177] Sigterm acknowledgement from replica 3 received [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:40.523576 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: I0725 23:19:40.645351 281472972651200 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: I0725 23:19:40.645569 281472972651200 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: I0725 23:19:40.646557 281472972651200 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: I0725 23:19:40.646601 281472972651200 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2h4zdtr__/tmp34hv_pgf/workertemp_2/fh_ckpt [worker-2]: I0725 23:19:40.692041 281472972651200 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2h4zdtr__/tmp34hv_pgf/workertemp_2/fh_ckpt [worker-2]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-2]: I0725 23:19:40.693586 281472972651200 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: I0725 23:19:40.693796 281472972651200 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2h4zdtr__/tmp34hv_pgf/workertemp_3/fh_ckpt [worker-1]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2h4zdtr__/tmp34hv_pgf/workertemp_1/fh_ckpt [worker-1]: I0725 23:19:40.712054 281472972651200 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2h4zdtr__/tmp34hv_pgf/workertemp_1/fh_ckpt [worker-0]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2h4zdtr__/tmp34hv_pgf/fh_ckpt [worker-0]: I0725 23:19:40.720566 281472972651200 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2h4zdtr__/tmp34hv_pgf/fh_ckpt [worker-0]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-0]: I0725 23:19:40.722036 281472972651200 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: I0725 23:19:40.722231 281472972651200 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-1]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-1]: I0725 23:19:40.727067 281472972651200 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-1]: I0725 23:19:40.727311 281472972651200 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: I0725 23:19:40.708892 281472972651200 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2h4zdtr__/tmp34hv_pgf/workertemp_3/fh_ckpt [worker-3]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-3]: I0725 23:19:40.749565 281472972651200 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: I0725 23:19:40.749826 281472972651200 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. INFO:tensorflow:restarting workers I0725 23:19:42.433436 281472975207104 failure_handler_test.py:309] restarting workers [worker-0]: I0725 23:19:42.484225 281472972651200 multi_process_runner.py:840] Subprocess with PID 932923 (worker, 0) is now being started. [worker-0]: I0725 23:19:42.484623 281472972651200 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38063", "localhost:42713", "localhost:38057", "localhost:34867"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' INFO:tensorflow:workers restarted I0725 23:19:42.526277 281472975207104 failure_handler_test.py:313] workers restarted [worker-1]: I0725 23:19:42.531458 281472972651200 multi_process_runner.py:840] Subprocess with PID 933032 (worker, 1) is now being started. [worker-1]: I0725 23:19:42.531871 281472972651200 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38063", "localhost:42713", "localhost:38057", "localhost:34867"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0725 23:19:42.558039 281472972651200 multi_process_runner.py:840] Subprocess with PID 933114 (worker, 2) is now being started. [worker-2]: I0725 23:19:42.558438 281472972651200 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38063", "localhost:42713", "localhost:38057", "localhost:34867"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-1]: 2023-07-25 23:19:42.599651: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:42713 [worker-3]: I0725 23:19:42.610692 281472972651200 multi_process_runner.py:840] Subprocess with PID 933383 (worker, 3) is now being started. [worker-3]: I0725 23:19:42.611099 281472972651200 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38063", "localhost:42713", "localhost:38057", "localhost:34867"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: 2023-07-25 23:19:42.647399: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:38057 [worker-0]: 2023-07-25 23:19:42.707826: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:38063 [worker-3]: 2023-07-25 23:19:42.729499: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:34867 [worker-0]: 2023-07-25 23:19:42.792090: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 15106628545906112861 [worker-2]: 2023-07-25 23:19:42.792999: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-1]: 2023-07-25 23:19:42.795966: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-07-25 23:19:42.795339: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 16495809836586618984 [worker-0]: 2023-07-25 23:19:42.795464: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 15580737163808572963 [worker-0]: 2023-07-25 23:19:42.795594: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 2178370914605920636 [worker-0]: 2023-07-25 23:19:42.796972: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: 2023-07-25 23:19:42.806693: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0725 23:19:42.808882 281472972651200 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0725 23:19:42.808589 281472972651200 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0725 23:19:42.817982 281472972651200 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0725 23:19:42.837760 281472972651200 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0725 23:19:42.862674 281472972651200 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0725 23:19:42.863387 281472972651200 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38063', 'localhost:42713', 'localhost:38057', 'localhost:34867']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0725 23:19:42.863646 281472972651200 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38063', 'localhost:42713', 'localhost:38057', 'localhost:34867']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0725 23:19:42.905835 281472972651200 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0725 23:19:42.906495 281472972651200 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38063', 'localhost:42713', 'localhost:38057', 'localhost:34867']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0725 23:19:42.906727 281472972651200 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38063', 'localhost:42713', 'localhost:38057', 'localhost:34867']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0725 23:19:42.948477 281472972651200 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0725 23:19:42.967084 281472972651200 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38063', 'localhost:42713', 'localhost:38057', 'localhost:34867']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0725 23:19:42.968024 281472972651200 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38063', 'localhost:42713', 'localhost:38057', 'localhost:34867']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0725 23:19:42.991623 281472972651200 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0725 23:19:42.992216 281472972651200 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38063', 'localhost:42713', 'localhost:38057', 'localhost:34867']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0725 23:19:42.992449 281472972651200 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38063', 'localhost:42713', 'localhost:38057', 'localhost:34867']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0725 23:19:43.205677 281472972651200 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0725 23:19:43.208372 281472972651200 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I0725 23:19:43.215407 281472972651200 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0725 23:19:43.215829 281472972651200 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0725 23:19:43.216689 281472972651200 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 24 [worker-2]: I0725 23:19:43.216895 281472972651200 failure_handler_test.py:197] Start training at 24 [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0725 23:19:43.224229 281472972651200 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0725 23:19:43.226039 281472972651200 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0725 23:19:43.226533 281472972651200 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0725 23:19:43.226925 281472972651200 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 24 [worker-1]: I0725 23:19:43.227119 281472972651200 failure_handler_test.py:197] Start training at 24 [worker-2]: INFO:tensorflow:training restarted [worker-2]: I0725 23:19:43.220907 281472972651200 failure_handler_test.py:207] training restarted [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0725 23:19:43.236532 281472972651200 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I0725 23:19:43.238645 281472972651200 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0725 23:19:43.239070 281472972651200 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0725 23:19:43.239484 281472972651200 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 24 [worker-0]: I0725 23:19:43.239680 281472972651200 failure_handler_test.py:197] Start training at 24 [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I0725 23:19:43.269218 281472972651200 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0725 23:19:43.269629 281472972651200 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0725 23:19:43.270020 281472972651200 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 24 [worker-3]: I0725 23:19:43.270216 281472972651200 failure_handler_test.py:197] Start training at 24 [worker-1]: INFO:tensorflow:training restarted [worker-1]: I0725 23:19:43.275779 281472972651200 failure_handler_test.py:207] training restarted [worker-0]: INFO:tensorflow:training restarted [worker-0]: I0725 23:19:43.308301 281472972651200 failure_handler_test.py:207] training restarted [worker-3]: INFO:tensorflow:training restarted [worker-3]: I0725 23:19:43.327802 281472972651200 failure_handler_test.py:207] training restarted [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:43.447669 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:43.498058 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:43.514852 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:43.612174 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:43.720017 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:43.719233 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:43.731179 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:43.771922 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:43.863568 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:43.844987 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:43.871627 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:43.880695 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:43.993989 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:44.007408 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:44.013741 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:43.991428 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:44.100904 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:44.104846 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:44.109297 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:44.117753 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8420ccc0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0725 23:19:44.208605 281472972651200 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8420ccc0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff84206840> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0725 23:19:44.208287 281472972651200 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff84206840> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8420bb00> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0725 23:19:44.226657 281472972651200 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8420bb00> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff842091c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0725 23:19:44.236814 281472972651200 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff842091c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:44.247026 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:44.236826 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:44.251424 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:44.250756 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff84207d80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0725 23:19:44.332981 281472972651200 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff84207d80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff84208ea0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 1 finished [worker-2]: W0725 23:19:44.333780 281472972651200 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff84208ea0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: I0725 23:19:44.333445 281472972651200 failure_handler_test.py:195] epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0725 23:19:44.334220 281472972651200 failure_handler_test.py:195] epoch 1 finished [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff842099e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0725 23:19:44.342603 281472972651200 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff842099e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0725 23:19:44.343059 281472972651200 failure_handler_test.py:195] epoch 1 finished [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8420f2e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:44.348357 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: W0725 23:19:44.346517 281472972651200 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8420f2e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0725 23:19:44.346955 281472972651200 failure_handler_test.py:195] epoch 1 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:44.358457 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:44.346849 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:44.371395 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:44.468988 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:44.469010 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:44.492640 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:44.483251 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:44.560464 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:44.574203 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:44.591167 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:44.601279 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:44.703708 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:44.715440 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:44.707594 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:44.735036 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:44.806308 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:44.816176 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:44.871134 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:44.873693 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:45.096220 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:45.091377 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:45.106580 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:45.073749 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:45.443186 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:45.436853 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:45.438883 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:45.438858 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:45.529493 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:45.536875 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:45.549948 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:45.575648 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:45.651602 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:45.650519 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:45.651558 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:45.712625 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:45.807078 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:45.817672 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:45.817769 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:45.813078 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:45.890298 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:45.917183 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:45.939831 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:45.938182 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:46.012449 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:46.016071 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:46.036082 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:46.031807 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:46.137189 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:46.132075 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:46.131641 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:46.161706 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:46.229750 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:46.229398 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:46.230790 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:46.229658 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:46.304049 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:46.308346 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:46.306176 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:46.304980 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I0725 23:19:46.365379 281472972651200 failure_handler_test.py:195] epoch 2 finished [worker-3]: I0725 23:19:46.365221 281472972651200 failure_handler_test.py:195] epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0725 23:19:46.368503 281472972651200 failure_handler_test.py:195] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0725 23:19:46.368933 281472972651200 failure_handler_test.py:195] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:46.375457 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:46.375194 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:46.378804 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:46.397331 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:46.498414 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:46.501850 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:46.502610 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:46.511114 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:46.606604 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:46.612251 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:46.602107 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:46.606896 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:46.671662 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:46.673099 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:46.677182 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:46.672152 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:46.797697 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:46.801764 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:46.810560 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:46.826927 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:46.956243 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:46.971382 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:47.001158 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:47.041213 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:47.143220 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:47.146798 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:47.152103 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:47.155526 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:47.213758 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:47.214009 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:47.213939 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:47.221213 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:47.281747 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:47.285813 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:47.290062 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:47.281778 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:47.376178 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:47.391333 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:47.401330 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:47.375060 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:47.565505 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:47.586140 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:47.588789 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:47.587597 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:47.703159 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:47.721362 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:47.721472 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:47.730326 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:47.813464 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:47.813576 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:47.815214 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:47.814739 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:47.910625 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:47.911449 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:47.915656 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:47.951850 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:48.010878 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:48.012137 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:48.011641 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:48.041692 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0725 23:19:48.107584 281472972651200 failure_handler_test.py:195] epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I0725 23:19:48.115781 281472972651200 failure_handler_test.py:195] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-1]: I0725 23:19:48.117013 281472972651200 failure_handler_test.py:195] epoch 3 finished [worker-2]: I0725 23:19:48.117388 281472972651200 failure_handler_test.py:195] epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:48.126930 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:48.125892 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:48.117822 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:48.141365 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:48.273855 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:48.297264 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:48.283224 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:48.320485 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:48.407502 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:48.407798 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:48.425278 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:48.407268 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:48.547302 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:48.558813 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:48.560763 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:48.547388 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:48.643741 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:48.646966 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:48.640928 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:48.661588 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:48.731159 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:48.733933 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:48.750848 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:48.757803 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:48.817342 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:48.820608 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:48.822039 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:48.822852 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:48.947677 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:48.951150 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:48.954430 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:48.957120 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:49.019635 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:49.019899 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:49.024762 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:49.027765 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:49.151409 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:49.150985 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:49.156545 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:49.156259 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:49.291532 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:49.299797 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:49.305491 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:49.303137 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:49.413759 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:49.416415 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:49.414324 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:49.420487 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:49.490264 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:49.490023 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:49.502738 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:49.517357 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:49.588017 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:49.591800 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:49.605825 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:49.623860 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:49.711177 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:49.715641 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:49.706084 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:49.711191 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0725 23:19:49.766431 281472972651200 failure_handler_test.py:195] epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I0725 23:19:49.776528 281472972651200 failure_handler_test.py:195] epoch 4 finished [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0725 23:19:49.776279 281472972651200 failure_handler_test.py:195] epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:49.776290 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:49.786238 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 4 finished [worker-0]: I0725 23:19:49.787803 281472972651200 failure_handler_test.py:195] epoch 4 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:49.804937 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:49.799005 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:49.883211 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:49.883976 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:49.901202 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:49.911338 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:49.994820 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:50.008456 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:50.017229 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:50.041318 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:50.180470 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:50.185992 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:50.190929 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:50.191903 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:50.313735 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:50.316475 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:50.345603 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:50.331306 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:50.435926 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:50.437795 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:50.436779 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:50.458384 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:50.536615 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:50.551432 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:50.545825 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:50.568909 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:50.657098 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:50.654532 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:50.674032 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:50.681449 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:50.800955 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:50.804544 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:50.797343 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:50.804763 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:50.932138 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:50.937279 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:50.961059 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:50.977305 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:51.042357 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:51.056194 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:51.067401 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:51.057850 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:51.166158 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:51.169477 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:51.170751 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:51.212400 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:51.294531 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:51.314109 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:51.330723 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:51.322533 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:51.446385 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:51.446998 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:51.453576 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:51.460264 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:51.568512 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:51.576080 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:51.579740 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:51.568450 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 5 finished [worker-0]: INFO:tensorflow:epoch 5 finished [worker-0]: I0725 23:19:51.660184 281472972651200 failure_handler_test.py:195] epoch 5 finished [worker-1]: I0725 23:19:51.660001 281472972651200 failure_handler_test.py:195] epoch 5 finished [worker-3]: INFO:tensorflow:epoch 5 finished [worker-3]: I0725 23:19:51.666557 281472972651200 failure_handler_test.py:195] epoch 5 finished [worker-2]: INFO:tensorflow:epoch 5 finished [worker-2]: I0725 23:19:51.672989 281472972651200 failure_handler_test.py:195] epoch 5 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:51.676468 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:51.685716 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:51.676513 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:51.691643 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:51.786275 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:51.786283 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:51.796320 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:51.818382 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:51.880066 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:51.878954 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:51.878192 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:51.921441 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:51.986526 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:52.006370 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:51.999696 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:52.024025 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:52.106693 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:52.107108 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:52.097806 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:52.141281 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:52.329085 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:52.340582 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:52.342367 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:52.340927 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:52.404273 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:52.410637 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:52.422967 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:52.418251 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:52.483959 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:52.484883 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:52.483663 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:52.528430 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:52.612003 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:52.598254 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:52.625847 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:52.625944 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:52.707566 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:52.714929 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:52.726540 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:52.727587 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:52.807067 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:52.810952 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:52.810998 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:52.840951 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I0725 23:19:54.516044 281472975207104 multi_process_runner.py:646] worker-0 exit code: 0 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I0725 23:19:59.043454 281472975207104 multi_process_runner.py:646] worker-1 exit code: 0 I0725 23:19:59.043719 281472975207104 multi_process_runner.py:646] worker-2 exit code: 0 I0725 23:19:59.043885 281472975207104 multi_process_runner.py:646] worker-3 exit code: 0 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.002249 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:53.003822 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.064551 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.004121 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I0725 23:19:59.047506 281472975207104 multi_process_runner.py:662] Joining log reading threads. [worker-1]: I0725 23:19:53.007618 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.064057 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:53.063505 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:53.064084 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.122246 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.122465 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:53.121671 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.179196 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:53.121754 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.180006 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 6 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:53.178948 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 6 finished [worker-3]: I0725 23:19:53.227723 281472972651200 failure_handler_test.py:195] epoch 6 finished [worker-0]: I0725 23:19:53.179843 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 6 finished [worker-0]: INFO:tensorflow:epoch 6 finished [worker-2]: I0725 23:19:53.228102 281472972651200 failure_handler_test.py:195] epoch 6 finished [worker-1]: I0725 23:19:53.227980 281472972651200 failure_handler_test.py:195] epoch 6 finished [worker-0]: I0725 23:19:53.227950 281472972651200 failure_handler_test.py:195] epoch 6 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.237785 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.237706 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:53.236693 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:53.238034 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.294177 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:53.294306 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.294628 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.353938 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:53.353995 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.412872 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.354274 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:53.295027 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.414261 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:53.413752 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.470054 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:53.354174 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.470025 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:53.413178 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.523821 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.524289 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:53.469212 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.577618 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:53.469846 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.577632 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:53.522654 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.631984 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:53.524080 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.631517 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:53.576364 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.685500 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.685180 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:53.577555 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:53.630469 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.738191 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:53.631723 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.737736 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:53.684482 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.790899 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.790496 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:53.736992 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.844060 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.844467 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:53.685311 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.897581 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.897793 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:53.789953 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:53.738010 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:53.843080 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:53.790812 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:53.950915 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:53.950514 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:53.896678 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:53.844256 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:19:54.003551 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:54.003254 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:19:53.897609 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 7 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:54.050577 281472972651200 failure_handler_test.py:195] epoch 7 finished [worker-0]: I0725 23:19:53.950842 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Training finished. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:19:54.051657 281472972651200 failure_handler_test.py:245] Training finished. [worker-0]: I0725 23:19:54.003322 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 7 finished [worker-0]: I0725 23:19:54.050701 281472972651200 failure_handler_test.py:195] epoch 7 finished [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I0725 23:19:54.052042 281472972651200 failure_handler_test.py:245] Training finished. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:53.949772 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 7 finished [worker-2]: I0725 23:19:54.050866 281472972651200 failure_handler_test.py:195] epoch 7 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:19:54.002695 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Training finished. [worker-1]: INFO:tensorflow:epoch 7 finished [worker-2]: I0725 23:19:54.052504 281472972651200 failure_handler_test.py:245] Training finished. [worker-1]: I0725 23:19:54.050782 281472972651200 failure_handler_test.py:195] epoch 7 finished [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I0725 23:19:54.051932 281472972651200 failure_handler_test.py:245] Training finished. I0725 23:19:59.087111 281472975207104 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker): 28.02s I0725 23:19:59.584959 281472975207104 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker): 28.02s [ OK ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 46093 I0725 23:19:59.586695 281472975207104 test_util.py:3813] Using local port 46093 INFO:tensorflow:Using local port 40383 I0725 23:19:59.587084 281472975207104 test_util.py:3813] Using local port 40383 INFO:tensorflow:Using local port 37249 I0725 23:19:59.587436 281472975207104 test_util.py:3813] Using local port 37249 INFO:tensorflow:Using local port 35759 I0725 23:19:59.587786 281472975207104 test_util.py:3813] Using local port 35759 INFO:tensorflow:Cluster starting. I0725 23:19:59.631000 281472975207104 failure_handler_test.py:297] Cluster starting. [worker-1]: I0725 23:19:59.754100 281472972651200 multi_process_runner.py:840] Subprocess with PID 951946 (worker, 1) is now being started. [worker-1]: I0725 23:19:59.754509 281472972651200 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:46093", "localhost:40383", "localhost:37249", "localhost:35759"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: I0725 23:19:59.850738 281472972651200 multi_process_runner.py:840] Subprocess with PID 951939 (worker, 0) is now being started. [worker-3]: I0725 23:19:59.851812 281472972651200 multi_process_runner.py:840] Subprocess with PID 952077 (worker, 3) is now being started. [worker-3]: I0725 23:19:59.852175 281472972651200 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:46093", "localhost:40383", "localhost:37249", "localhost:35759"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: I0725 23:19:59.851134 281472972651200 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:46093", "localhost:40383", "localhost:37249", "localhost:35759"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: I0725 23:19:59.859291 281472972651200 multi_process_runner.py:840] Subprocess with PID 952004 (worker, 2) is now being started. [worker-2]: I0725 23:19:59.859683 281472972651200 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:46093", "localhost:40383", "localhost:37249", "localhost:35759"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-2]: 2023-07-25 23:19:59.893999: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:37249 [worker-0]: 2023-07-25 23:19:59.896389: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:46093 [worker-0]: 2023-07-25 23:19:59.905550: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 3578262714712425734 [worker-3]: 2023-07-25 23:19:59.905712: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:35759 [worker-2]: 2023-07-25 23:19:59.905932: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-07-25 23:19:59.909783: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 8799691235254083873 [worker-0]: 2023-07-25 23:19:59.910020: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-07-25 23:19:59.914370: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 9291278116937310831 [worker-3]: 2023-07-25 23:19:59.914625: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-1]: 2023-07-25 23:19:59.915591: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:40383 [worker-0]: 2023-07-25 23:19:59.920359: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 1600879306995955312 [worker-1]: 2023-07-25 23:19:59.920618: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0725 23:19:59.922543 281472972651200 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: I0725 23:19:59.922524 281472972651200 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0725 23:19:59.922612 281472972651200 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0725 23:19:59.932328 281472972651200 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-3]: I0725 23:19:59.979482 281472972651200 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-2]: I0725 23:19:59.979480 281472972651200 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-1]: I0725 23:19:59.979548 281472972651200 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-2]: INFO:tensorflow:Check health not enabled. [worker-1]: INFO:tensorflow:Check health not enabled. [worker-3]: I0725 23:19:59.980027 281472972651200 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: I0725 23:19:59.980047 281472972651200 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46093', 'localhost:40383', 'localhost:37249', 'localhost:35759']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0725 23:19:59.980098 281472972651200 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46093', 'localhost:40383', 'localhost:37249', 'localhost:35759']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0725 23:19:59.980259 281472972651200 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46093', 'localhost:40383', 'localhost:37249', 'localhost:35759']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46093', 'localhost:40383', 'localhost:37249', 'localhost:35759']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0725 23:19:59.980277 281472972651200 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46093', 'localhost:40383', 'localhost:37249', 'localhost:35759']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0725 23:19:59.980332 281472972651200 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46093', 'localhost:40383', 'localhost:37249', 'localhost:35759']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0725 23:19:59.987469 281472972651200 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0725 23:19:59.988175 281472972651200 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46093', 'localhost:40383', 'localhost:37249', 'localhost:35759']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0725 23:19:59.988419 281472972651200 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46093', 'localhost:40383', 'localhost:37249', 'localhost:35759']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0725 23:20:00.038118 281472972651200 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: I0725 23:20:00.038148 281472972651200 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: I0725 23:20:00.038443 281472972651200 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I0725 23:20:00.038020 281472972651200 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: I0725 23:20:00.039273 281472972651200 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0725 23:20:00.038820 281472972651200 failure_handling.py:674] Start watcher for local signal. [worker-1]: I0725 23:20:00.039225 281472972651200 failure_handling.py:674] Start watcher for local signal. [worker-3]: I0725 23:20:00.039530 281472972651200 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: I0725 23:20:00.038756 281472972651200 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0725 23:20:00.039061 281472972651200 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: Instructions for updating: [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: I0725 23:20:00.039464 281472972651200 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: I0725 23:20:00.039008 281472972651200 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Instructions for updating: [worker-3]: W0725 23:20:00.039853 281472972651200 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0725 23:20:00.039367 281472972651200 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Instructions for updating: [worker-3]: INFO:tensorflow:Start training at 0 [worker-1]: W0725 23:20:00.039785 281472972651200 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: I0725 23:20:00.040038 281472972651200 failure_handler_test.py:197] Start training at 0 [worker-1]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: I0725 23:20:00.039553 281472972651200 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Start training at 0 [worker-0]: W0725 23:20:00.039350 281472972651200 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: I0725 23:20:00.039969 281472972651200 failure_handler_test.py:197] Start training at 0 [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0725 23:20:00.039538 281472972651200 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:00.206559 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:00.208258 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:00.210661 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:00.211612 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending sigterm [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I0725 23:20:01.831276 281472975207104 failure_handler_test.py:302] sending sigterm [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:00.280354 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:00.280350 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [ FAILED ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:00.281623 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker): 15.48s [worker-1]: I0725 23:20:00.281252 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I0725 23:20:15.066614 281472975207104 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker): 15.48s ====================================================================== [worker-0]: I0725 23:20:00.342061 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 ERROR: test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker (__main__.PreemptionCheckpointTest) PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker(api_wrapping_train=True, input_arg='checkpoint', strategy_option='MWMS_multi_worker')[worker-2]: I0725 23:20:00.341033 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 ---------------------------------------------------------------------- [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 304, in test_preemption_checkpointing os.kill(mpr.get_process_id('worker', killed_worker), signal.SIGTERM) ProcessLookupError: [Errno 3] No such process [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 ---------------------------------------------------------------------- Ran 3 tests in 47.553s [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 FAILED (errors=1) [worker-3]: I0725 23:20:00.340910 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:00.342396 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:00.401796 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:00.400006 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:00.399981 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:00.401714 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:00.461760 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:00.460907 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff84200180> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: I0725 23:20:00.460898 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff84200860> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: I0725 23:20:00.461753 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: W0725 23:20:00.510912 281472972651200 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff84200180> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff84200860> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0725 23:20:00.511037 281472972651200 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff84200860> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff84200860> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: W0725 23:20:00.510700 281472972651200 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff84200860> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0725 23:20:00.511059 281472972651200 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff84200860> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:00.521029 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:00.519433 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff842023e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: I0725 23:20:00.519410 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:00.521223 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff84200b80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0725 23:20:00.572096 281472972651200 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff842023e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff84200b80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff84200720> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: W0725 23:20:00.572203 281472972651200 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff84200b80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0725 23:20:00.572228 281472972651200 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff84200720> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0725 23:20:00.571842 281472972651200 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff84200b80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: I0725 23:20:00.581727 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:00.580523 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:00.581728 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:00.641714 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:00.580540 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:00.639754 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:00.699827 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:00.641036 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:00.639704 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:00.698549 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:00.756453 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:00.699821 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:00.698314 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:00.756393 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:00.810167 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:00.756311 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:00.757334 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:00.809771 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:00.809531 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:00.810312 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:00.863823 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:00.863937 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:00.863588 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:00.863257 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.041519 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.041110 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.040903 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.093733 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.092873 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.092450 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.041476 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.145504 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.144826 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.145201 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-3]: I0725 23:20:01.189713 281472972651200 failure_handler_test.py:195] epoch 0 finished [worker-0]: I0725 23:20:01.093622 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.190096 281472972651200 failure_handler_test.py:195] epoch 0 finished [worker-2]: I0725 23:20:01.190090 281472972651200 failure_handler_test.py:195] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.196973 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.145393 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.197691 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.198887 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.189943 281472972651200 failure_handler_test.py:195] epoch 0 finished [worker-3]: I0725 23:20:01.250354 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.251394 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.250718 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.198591 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.303815 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.302825 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.303154 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.251383 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.356724 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.356373 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.356400 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.304124 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.408966 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.408039 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.408353 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.357000 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.460482 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.460039 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.460448 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.511334 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.408747 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.512200 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.511734 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.459916 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.563163 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.563470 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.563584 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.512169 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.614077 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.614692 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.614381 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.668186 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.668233 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.563022 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.719365 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.667623 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.719714 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.614844 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.718986 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.771007 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.667586 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.770991 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.770257 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.719570 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.822241 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.822442 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.821954 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.770442 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.873363 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.873065 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.872568 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.822102 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.923346 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.923579 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 1 finished [worker-3]: I0725 23:20:01.923040 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:01.967233 281472972651200 failure_handler_test.py:195] epoch 1 finished [worker-1]: I0725 23:20:01.967203 281472972651200 failure_handler_test.py:195] epoch 1 finished [worker-0]: I0725 23:20:01.872790 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.966870 281472972651200 failure_handler_test.py:195] epoch 1 finished [worker-2]: I0725 23:20:01.974757 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.923169 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:01.974727 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:01.974035 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.967038 281472972651200 failure_handler_test.py:195] epoch 1 finished [worker-2]: I0725 23:20:02.024961 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.025111 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:01.974215 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.024607 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.077349 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.077464 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.024780 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.129190 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.076639 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.129779 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.076739 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.181399 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.128886 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.232323 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.181301 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.129081 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.285479 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.180608 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.232684 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.180780 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.335952 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.231982 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.285221 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.387353 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.284732 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.232146 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.336231 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.437339 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.284898 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.335632 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.488241 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.386595 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.335813 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.538410 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.437009 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.387093 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.386762 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.589443 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.487460 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.437575 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.437162 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.639219 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.487955 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.538131 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.487692 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.689895 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 2 finished [worker-3]: I0725 23:20:02.588593 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.538308 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.538632 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.733496 281472972651200 failure_handler_test.py:195] epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.638843 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.588793 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.589004 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.740725 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.689419 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.638994 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.639292 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.791094 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.689553 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.733130 281472972651200 failure_handler_test.py:195] epoch 2 finished [worker-2]: I0725 23:20:02.841410 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.690016 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.733294 281472972651200 failure_handler_test.py:195] epoch 2 finished [worker-2]: I0725 23:20:02.893127 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 2 finished [worker-3]: I0725 23:20:02.740410 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.733466 281472972651200 failure_handler_test.py:195] epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.740482 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.790668 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.943536 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.740969 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.790818 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.841027 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:02.994416 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.790917 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.841311 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.892714 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.044678 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.841456 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.892810 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.943159 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.095359 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.893123 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.943311 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:02.993979 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.145751 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.943783 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:02.994147 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.044297 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.195809 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:02.994368 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.044477 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.094987 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.245983 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.095054 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.044800 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.145287 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.296007 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.145504 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.095315 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.195489 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.346285 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.195611 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.145770 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.245650 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.396285 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.245832 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.195837 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.295593 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.446433 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.295731 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.245999 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 3 finished [worker-3]: I0725 23:20:03.345864 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.491286 281472972651200 failure_handler_test.py:195] epoch 3 finished [worker-0]: I0725 23:20:03.346008 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.295978 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.395966 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.498472 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.396186 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.346281 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.446152 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.548612 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-0]: I0725 23:20:03.446214 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.396324 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.490950 281472972651200 failure_handler_test.py:195] epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.598636 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.491099 281472972651200 failure_handler_test.py:195] epoch 3 finished [worker-1]: I0725 23:20:03.446589 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.498185 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 3 finished [worker-2]: I0725 23:20:03.648806 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.498275 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.491257 281472972651200 failure_handler_test.py:195] epoch 3 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.548312 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.699363 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.548455 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.498665 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.598264 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.749485 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.598409 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.548664 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.648438 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.799657 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.648619 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.598649 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.698973 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.849835 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.699177 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.648825 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.749149 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.900276 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.749303 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.699369 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.799208 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:03.950447 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.799342 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.749606 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.849508 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.000504 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.849643 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.799614 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.900022 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.050520 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.900144 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.849976 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:03.950098 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.102244 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:03.950334 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.900400 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.000214 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.153548 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.000326 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:03.950595 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.050223 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.204559 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.050331 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.000555 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 4 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.101838 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.247837 281472972651200 failure_handler_test.py:195] epoch 4 finished [worker-0]: I0725 23:20:04.102013 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.050578 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.153142 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.255108 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.102147 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.153375 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.305165 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.204261 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.154245 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.204384 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 4 finished [worker-2]: I0725 23:20:04.355108 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.247495 281472972651200 failure_handler_test.py:195] epoch 4 finished [worker-0]: I0725 23:20:04.247657 281472972651200 failure_handler_test.py:195] epoch 4 finished [worker-1]: I0725 23:20:04.204668 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 4 finished [worker-2]: I0725 23:20:04.405115 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.254923 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.254966 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.247825 281472972651200 failure_handler_test.py:195] epoch 4 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.455131 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.304890 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.304994 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.255332 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.505021 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.354930 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.354753 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.305254 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.555141 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.404901 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.404716 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.355230 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.605048 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.454978 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.454801 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.405102 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.655022 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.504856 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.504731 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.455230 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.706356 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.505164 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.554795 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.554925 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.756408 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.555185 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.604732 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.604850 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.806397 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.605072 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.654696 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.654865 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.856271 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.655040 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.706019 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.706163 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.706365 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.906197 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.756047 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.756215 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.756442 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.956134 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.806200 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 5 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.805970 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.806321 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.856045 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:04.999267 281472972651200 failure_handler_test.py:195] epoch 5 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.856322 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.905984 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.855907 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.006615 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.906179 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.955940 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.905867 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.056285 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 5 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.956192 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:04.999096 281472972651200 failure_handler_test.py:195] epoch 5 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.955784 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 5 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 5 finished [worker-2]: I0725 23:20:05.106392 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:04.999234 281472972651200 failure_handler_test.py:195] epoch 5 finished [worker-0]: I0725 23:20:05.006293 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:04.998915 281472972651200 failure_handler_test.py:195] epoch 5 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.156534 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.006501 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.006094 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.056058 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.206528 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.055874 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.056350 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.106214 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.256515 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.106122 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.106613 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.156352 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.307938 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.156214 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.156614 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.206392 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.357829 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.206231 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.206596 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.256253 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.407885 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.256092 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.256551 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.307701 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.457613 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.307518 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.307925 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.357694 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.357550 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.507624 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.357894 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.407611 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.407517 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.557461 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.407853 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.457283 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.457484 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.607374 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.457686 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.507273 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.507420 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.657441 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.507572 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.557091 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.707560 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.557263 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.557514 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 6 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.607035 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.750689 281472972651200 failure_handler_test.py:195] epoch 6 finished [worker-0]: I0725 23:20:05.607165 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.607423 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.657099 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.757852 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.657241 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.657436 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.707225 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.808018 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.707376 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 6 finished [worker-1]: I0725 23:20:05.707468 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.750331 281472972651200 failure_handler_test.py:195] epoch 6 finished [worker-0]: INFO:tensorflow:epoch 6 finished [worker-2]: I0725 23:20:05.858003 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 6 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.750508 281472972651200 failure_handler_test.py:195] epoch 6 finished [worker-1]: I0725 23:20:05.750678 281472972651200 failure_handler_test.py:195] epoch 6 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.757537 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.909385 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.757695 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.758165 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.807725 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:05.959438 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.807914 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.808106 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:06.009416 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.857718 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.857846 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.858031 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:06.059377 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.909095 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.909238 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.909466 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:06.109423 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:05.959113 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:05.959219 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:05.959583 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:06.159381 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:06.009129 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:06.009275 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:06.009536 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:06.209426 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:06.059250 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:06.059037 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:06.059416 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:06.259297 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:06.109229 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:06.109059 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:06.109445 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:06.309169 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:06.159194 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:06.159097 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:06.159434 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:06.359136 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:06.209213 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:06.209055 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:06.209455 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:06.409083 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:06.259097 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:06.258929 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:06.259310 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:06.308959 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:06.308865 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:06.459081 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:06.309216 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 7 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:06.358720 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:06.503729 281472972651200 failure_handler_test.py:195] epoch 7 finished [worker-1]: I0725 23:20:06.359169 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:06.358929 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Training finished. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:06.408753 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:20:06.504623 281472972651200 failure_handler_test.py:245] Training finished. [worker-1]: I0725 23:20:06.409110 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:20:06.408884 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:20:06.458758 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:20:06.459105 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 7 finished [worker-0]: I0725 23:20:06.458915 281472972651200 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 7 finished [worker-3]: I0725 23:20:06.503378 281472972651200 failure_handler_test.py:195] epoch 7 finished [worker-0]: INFO:tensorflow:epoch 7 finished [worker-1]: I0725 23:20:06.503706 281472972651200 failure_handler_test.py:195] epoch 7 finished [worker-3]: INFO:tensorflow:Training finished. [worker-0]: I0725 23:20:06.503551 281472972651200 failure_handler_test.py:195] epoch 7 finished [worker-1]: INFO:tensorflow:Training finished. [worker-3]: I0725 23:20:06.504048 281472972651200 failure_handler_test.py:245] Training finished. [worker-0]: INFO:tensorflow:Training finished. [worker-1]: I0725 23:20:06.504747 281472972651200 failure_handler_test.py:245] Training finished. [worker-0]: I0725 23:20:06.504483 281472972651200 failure_handler_test.py:245] Training finished. ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:multiple_return_tuple.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/multiple_return_tuple.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:multiple_return_tuple.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/multiple_return_tuple.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:multiple_return_tuple.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/multiple_return_tuple.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:sharding.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/sharding.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:sharding.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/sharding.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:sharding.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/sharding.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-constant.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-op-with-constant.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-constant.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-op-with-constant.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-constant.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-op-with-constant.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_conv.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_conv.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_conv.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_conv.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_conv.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_conv.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:no-bug.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/no-bug.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:no-bug.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/no-bug.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:no-bug.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/no-bug.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:inline-scf-while.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/inline-scf-while.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:inline-scf-while.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/inline-scf-while.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:inline-scf-while.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/inline-scf-while.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:compilation_pipeline.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/compilation_pipeline.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:compilation_pipeline.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/compilation_pipeline.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:compilation_pipeline.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/compilation_pipeline.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_abi_legalization.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_abi_legalization.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_abi_legalization.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_abi_legalization.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_abi_legalization.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_abi_legalization.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:frontend_attributes.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/frontend_attributes.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:frontend_attributes.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/frontend_attributes.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:frontend_attributes.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/frontend_attributes.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cublas_lt_matmul.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_cublas_lt_matmul.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cublas_lt_matmul.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_cublas_lt_matmul.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cublas_lt_matmul.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_cublas_lt_matmul.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:erase-op-without-results.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/erase-op-without-results.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:erase-op-without-results.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/erase-op-without-results.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:erase-op-without-results.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/erase-op-without-results.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:export_functions.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/export_functions.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:export_functions.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/export_functions.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:export_functions.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/export_functions.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops_verify.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/ir/tests/ops_verify.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops_verify.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/ir/tests/ops_verify.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops_verify.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/ir/tests/ops_verify.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:collective_ops.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/collective_ops.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:collective_ops.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/collective_ops.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:collective_ops.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/collective_ops.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-value.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-op-with-value.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-value.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-op-with-value.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-value.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-op-with-value.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:fft.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/fft.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:fft.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/fft.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:fft.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/fft.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:if_conditional.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/if_conditional.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:if_conditional.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/if_conditional.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:if_conditional.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/if_conditional.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_array.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_array.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_array.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_array.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_array.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_array.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/python/distribute/failure_handling:failure_handler_test (shard 1 of 8): Running tests under Python 3.11.1: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/python_aarch64-unknown-linux-gnu/bin/python3 [ RUN ] PreemptionCheckpointTest.test_error_propagation INFO:tensorflow:Using local port 44215 I0725 23:19:27.610653 281473054833344 test_util.py:3813] Using local port 44215 INFO:tensorflow:Using local port 42029 I0725 23:19:27.611289 281473054833344 test_util.py:3813] Using local port 42029 INFO:tensorflow:Using local port 38027 I0725 23:19:27.611676 281473054833344 test_util.py:3813] Using local port 38027 INFO:tensorflow:Using local port 37563 I0725 23:19:27.612055 281473054833344 test_util.py:3813] Using local port 37563 INFO:tensorflow:Cluster starting. I0725 23:19:32.782479 281473054833344 failure_handler_test.py:387] Cluster starting. [worker-0]: I0725 23:19:32.817134 281473590852288 multi_process_runner.py:840] Subprocess with PID 924118 (worker, 0) is now being started. [worker-0]: I0725 23:19:32.817491 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:44215", "localhost:42029", "localhost:38027", "localhost:37563"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: I0725 23:19:32.873543 281473590852288 multi_process_runner.py:840] Subprocess with PID 924283 (worker, 2) is now being started. [worker-0]: E0725 23:19:32.977257293 924118 server_chttp2.cc:40] {"created":"@1690327172.977139685","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1690327172.977134250","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1690327172.977105037","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1690327172.977093691","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1690327172.977133090","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1690327172.977126214","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-0]: 2023-07-25 23:19:32.977413: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:608] UNKNOWN: Could not start gRPC server [worker-0]: 2023-07-25 23:19:32.978712: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:780] Could not start gRPC server [worker-1]: I0725 23:19:32.997804 281473590852288 multi_process_runner.py:840] Subprocess with PID 924263 (worker, 1) is now being started. [worker-2]: I0725 23:19:32.873901 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:44215", "localhost:42029", "localhost:38027", "localhost:37563"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-1]: I0725 23:19:32.998179 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:44215", "localhost:42029", "localhost:38027", "localhost:37563"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: Process _Process-2: [worker-3]: I0725 23:19:33.033923 281473590852288 multi_process_runner.py:840] Subprocess with PID 925757 (worker, 3) is now being started. [worker-3]: I0725 23:19:33.034287 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:44215", "localhost:42029", "localhost:38027", "localhost:37563"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: Traceback (most recent call last): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: self.run() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-0]: return self._actual_run() [worker-0]: ^^^^^^^^^^^^^^^^^^ [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-0]: app.run(lambda _: self._run_impl()) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-0]: _run_main(main, args) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-0]: sys.exit(main(argv)) [worker-0]: ^^^^^^^^^^ [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-0]: app.run(lambda _: self._run_impl()) [worker-0]: ^^^^^^^^^^^^^^^^ [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 108, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-0]: six.reraise(*info.exc_info) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-0]: raise value [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-0]: return_value = fn(*args, **kwargs) [worker-0]: ^^^^^^^^^^^^^^^^^^^ [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-0]: self.worker_fn( [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 146, in worker_fn [worker-0]: strategy = collective_all_reduce_strategy.CollectiveAllReduceStrategy() [worker-0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 186, in __init__ [worker-0]: CollectiveAllReduceExtended( [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 339, in __init__ [worker-0]: self._initialize_strategy(self._cluster_resolver, devices=devices) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 358, in _initialize_strategy [worker-0]: self._initialize_multi_worker(cluster_resolver) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 530, in _initialize_multi_worker [worker-0]: context.context().ensure_initialized() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized [worker-0]: pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) [worker-0]: tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server [worker-3]: 2023-07-25 23:19:33.102244: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:37563 [worker-2]: 2023-07-25 23:19:33.207035: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:38027 [worker-1]: 2023-07-25 23:19:33.462064: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:42029 E0725 23:23:43.018256 281473054833344 multi_process_runner.py:626] Timeout when joining for child processes. Terminating... I0725 23:23:43.018651 281473054833344 multi_process_runner.py:715] worker-0 has already exited. Not terminating. I0725 23:23:43.018972 281473054833344 multi_process_runner.py:721] worker-1 terminated with signal . I0725 23:23:43.019260 281473054833344 multi_process_runner.py:721] worker-2 terminated with signal . I0725 23:23:43.019479 281473054833344 multi_process_runner.py:721] worker-3 terminated with signal . [worker-1]: 2023-07-25 23:23:43.076820: W tensorflow/tsl/distributed_runtime/preemption/preemption_notifier.cc:89] SIGTERM caught at 2023-07-25T23:23:43.076733097+00:00 [worker-1]: 2023-07-25 23:23:43.082241: I tensorflow/core/common_runtime/eager/context_distributed_manager.cc:821] Preemption not exported to coordination service: UNAVAILABLE: failed to connect to all addresses [worker-1]: Additional GRPC error information from remote target /job:worker/replica:0/task:0 while calling /tensorflow.CoordinationService/InsertKeyValue: [worker-1]: :{"created":"@1690327423.082072076","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3940,"referenced_errors":[{"created":"@1690327423.082067941","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":392,"grpc_status":14}]} [worker-3]: 2023-07-25 23:23:43.856297: W tensorflow/tsl/distributed_runtime/preemption/preemption_notifier.cc:89] SIGTERM caught at 2023-07-25T23:23:43.856224112+00:00 [worker-3]: 2023-07-25 23:23:43.856532: I tensorflow/core/common_runtime/eager/context_distributed_manager.cc:821] Preemption not exported to coordination service: UNAVAILABLE: failed to connect to all addresses [worker-3]: Additional GRPC error information from remote target /job:worker/replica:0/task:0 while calling /tensorflow.CoordinationService/InsertKeyValue: [worker-3]: :{"created":"@1690327423.856391454","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3940,"referenced_errors":[{"created":"@1690327423.656257524","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":392,"grpc_status":14}]} [worker-2]: 2023-07-25 23:23:44.006548: W tensorflow/tsl/distributed_runtime/preemption/preemption_notifier.cc:89] SIGTERM caught at 2023-07-25T23:23:44.006464567+00:00 [worker-2]: 2023-07-25 23:23:44.326739: I tensorflow/core/common_runtime/eager/context_distributed_manager.cc:821] Preemption not exported to coordination service: UNAVAILABLE: failed to connect to all addresses [worker-2]: Additional GRPC error information from remote target /job:worker/replica:0/task:0 while calling /tensorflow.CoordinationService/InsertKeyValue: [worker-2]: :{"created":"@1690327424.326277192","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3940,"referenced_errors":[{"created":"@1690327424.326272362","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":392,"grpc_status":14}]} E0725 23:24:13.026271 281473054833344 multi_process_runner.py:632] Timeout when waiting for child processes to print stacktrace. Sending SIGKILL... I0725 23:24:13.026643 281473054833344 multi_process_runner.py:715] worker-0 has already exited. Not terminating. I0725 23:24:13.027446 281473054833344 multi_process_runner.py:721] worker-1 terminated with signal . I0725 23:24:13.028230 281473054833344 multi_process_runner.py:721] worker-2 terminated with signal . I0725 23:24:13.028901 281473054833344 multi_process_runner.py:721] worker-3 terminated with signal . [ FAILED ] PreemptionCheckpointTest.test_error_propagation INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_error_propagation): 286.0s I0725 23:24:13.611714 281473054833344 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_error_propagation): 286.0s [ RUN ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice INFO:tensorflow:Start watcher for local signal. I0725 23:24:13.697508 281473054833344 failure_handling.py:674] Start watcher for local signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. I0725 23:24:13.697863 281473054833344 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. W0725 23:24:13.698168 281473054833344 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. INFO:tensorflow:Start training at 0 I0725 23:24:13.698365 281473054833344 failure_handler_test.py:197] Start training at 0 WARNING:tensorflow:5 out of the last 5 calls to .distributed_train_step..train_step at 0xfffee0731ee0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0725 23:24:13.890816 281473054833344 polymorphic_function.py:156] 5 out of the last 5 calls to .distributed_train_step..train_step at 0xfffee0731ee0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. WARNING:tensorflow:6 out of the last 6 calls to .distributed_train_step..train_step at 0xfffee0731ee0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0725 23:24:13.906418 281473054833344 polymorphic_function.py:156] 6 out of the last 6 calls to .distributed_train_step..train_step at 0xfffee0731ee0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. INFO:tensorflow:epoch 0 finished I0725 23:24:14.044325 281473054833344 failure_handler_test.py:195] epoch 0 finished INFO:tensorflow:epoch 1 finished I0725 23:24:14.267247 281473054833344 failure_handler_test.py:195] epoch 1 finished INFO:tensorflow:epoch 2 finished I0725 23:24:14.471467 281473054833344 failure_handler_test.py:195] epoch 2 finished INFO:tensorflow:epoch 3 finished I0725 23:24:15.142910 281473054833344 failure_handler_test.py:195] epoch 3 finished INFO:tensorflow:epoch 4 finished I0725 23:24:15.564531 281473054833344 failure_handler_test.py:195] epoch 4 finished INFO:tensorflow:epoch 5 finished I0725 23:24:15.809635 281473054833344 failure_handler_test.py:195] epoch 5 finished INFO:tensorflow:epoch 6 finished I0725 23:24:16.109493 281473054833344 failure_handler_test.py:195] epoch 6 finished INFO:tensorflow:epoch 7 finished I0725 23:24:16.445743 281473054833344 failure_handler_test.py:195] epoch 7 finished INFO:tensorflow:Training finished. I0725 23:24:16.446429 281473054833344 failure_handler_test.py:245] Training finished. INFO:tensorflow:sending sigterm I0725 23:24:16.636215 281470143361504 failure_handler_test.py:467] sending sigterm INFO:tensorflow:Member single_worker has received termination notice. I0725 23:24:16.636798 281473054833344 failure_handling.py:701] Member single_worker has received termination notice. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice): 3.02s I0725 23:24:16.637346 281473054833344 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice): 3.02s [ OK ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 46359 I0725 23:24:16.639660 281473054833344 test_util.py:3813] Using local port 46359 INFO:tensorflow:Using local port 41815 I0725 23:24:16.640056 281473054833344 test_util.py:3813] Using local port 41815 INFO:tensorflow:Using local port 46379 I0725 23:24:16.640413 281473054833344 test_util.py:3813] Using local port 46379 INFO:tensorflow:Using local port 34407 I0725 23:24:16.640774 281473054833344 test_util.py:3813] Using local port 34407 INFO:tensorflow:Cluster starting. I0725 23:24:16.663451 281473054833344 failure_handler_test.py:297] Cluster starting. [worker-1]: I0725 23:24:16.744201 281473590852288 multi_process_runner.py:840] Subprocess with PID 1369004 (worker, 1) is now being started. [worker-1]: I0725 23:24:16.744607 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:46359", "localhost:41815", "localhost:46379", "localhost:34407"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: I0725 23:24:16.810266 281473590852288 multi_process_runner.py:840] Subprocess with PID 1368985 (worker, 0) is now being started. [worker-0]: I0725 23:24:16.810699 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:46359", "localhost:41815", "localhost:46379", "localhost:34407"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: I0725 23:24:16.812665 281473590852288 multi_process_runner.py:840] Subprocess with PID 1369022 (worker, 2) is now being started. [worker-2]: I0725 23:24:16.813056 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:46359", "localhost:41815", "localhost:46379", "localhost:34407"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0725 23:24:16.889185 281473590852288 multi_process_runner.py:840] Subprocess with PID 1369031 (worker, 3) is now being started. [worker-3]: I0725 23:24:16.889600 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:46359", "localhost:41815", "localhost:46379", "localhost:34407"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2023-07-25 23:24:16.925894: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:46359 [worker-1]: 2023-07-25 23:24:16.944698: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:41815 [worker-0]: 2023-07-25 23:24:16.970252: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 12737167581073804259 [worker-1]: 2023-07-25 23:24:16.970653: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-07-25 23:24:16.970440: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 8220783362643356875 [worker-0]: 2023-07-25 23:24:16.986353: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: 2023-07-25 23:24:17.008205: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:34407 [worker-0]: 2023-07-25 23:24:17.026180: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 6159056568707082600 [worker-3]: 2023-07-25 23:24:17.027194: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-2]: 2023-07-25 23:24:17.157312: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:46379 [worker-0]: 2023-07-25 23:24:17.160737: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 17415196037181725140 [worker-2]: 2023-07-25 23:24:17.161165: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0725 23:24:17.177200 281473590852288 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: I0725 23:24:17.196741 281473590852288 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0725 23:24:17.207175 281473590852288 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0725 23:24:17.251183 281473590852288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0725 23:24:17.251745 281473590852288 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46359', 'localhost:41815', 'localhost:46379', 'localhost:34407']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0725 23:24:17.251977 281473590852288 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46359', 'localhost:41815', 'localhost:46379', 'localhost:34407']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0725 23:24:17.257161 281473590852288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0725 23:24:17.257742 281473590852288 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46359', 'localhost:41815', 'localhost:46379', 'localhost:34407']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0725 23:24:17.257972 281473590852288 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46359', 'localhost:41815', 'localhost:46379', 'localhost:34407']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0725 23:24:17.235959 281473590852288 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0725 23:24:17.337257 281473590852288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0725 23:24:17.337994 281473590852288 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46359', 'localhost:41815', 'localhost:46379', 'localhost:34407']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0725 23:24:17.338245 281473590852288 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46359', 'localhost:41815', 'localhost:46379', 'localhost:34407']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0725 23:24:17.368733 281473590852288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0725 23:24:17.369280 281473590852288 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46359', 'localhost:41815', 'localhost:46379', 'localhost:34407']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0725 23:24:17.369509 281473590852288 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46359', 'localhost:41815', 'localhost:46379', 'localhost:34407']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0725 23:24:17.436952 281473590852288 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0725 23:24:17.441034 281473590852288 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0725 23:24:17.453809 281473590852288 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0725 23:24:17.456734 281473590852288 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0725 23:24:17.457067 281473590852288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0725 23:24:17.457370 281473590852288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I0725 23:24:17.466745 281473590852288 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0725 23:24:17.467112 281473590852288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0725 23:24:17.467423 281473590852288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0725 23:24:17.467620 281473590852288 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0725 23:24:17.457564 281473590852288 failure_handler_test.py:197] Start training at 0 [worker-3]: I0725 23:24:17.472049 281473590852288 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: I0725 23:24:17.456734 281473590852288 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0725 23:24:17.457083 281473590852288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0725 23:24:17.457385 281473590852288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0725 23:24:17.457575 281473590852288 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I0725 23:24:17.487063 281473590852288 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0725 23:24:17.487411 281473590852288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0725 23:24:17.487720 281473590852288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0725 23:24:17.487907 281473590852288 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:17.943170 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:18.004661 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:18.049128 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:18.180076 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:18.427910 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:18.435271 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:18.471019 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:18.540762 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:18.677806 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:18.684480 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:18.700656 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:18.700817 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:18.851001 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:18.870713 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:18.880687 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:18.880619 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:19.070706 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:19.100740 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:19.110301 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:19.140428 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffa8f91ee0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0725 23:24:19.307449 281473590852288 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffa8f91ee0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffa8f92ac0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0725 23:24:19.316840 281473590852288 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffa8f92ac0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffa8f94180> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0725 23:24:19.326810 281473590852288 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffa8f94180> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffa8f9a840> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0725 23:24:19.326395 281473590852288 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffa8f9a840> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:19.340468 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:19.340454 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:19.350601 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:19.377349 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9b060> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0725 23:24:19.517136 281473590852288 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9b060> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:19.527033 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffa8f91b20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffa8f94ea0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0725 23:24:19.562717 281473590852288 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffa8f94ea0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0725 23:24:19.557294 281473590852288 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffa8f91b20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffa8f932e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0725 23:24:19.566512 281473590852288 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffa8f932e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:19.582512 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:19.576542 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:19.600523 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:19.680784 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:19.690274 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:19.701031 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:19.740436 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:19.935355 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:19.949277 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:19.960257 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:19.961584 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:20.050531 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:20.070446 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:20.070483 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:20.131565 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:20.240682 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:20.260929 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:20.280545 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:20.297042 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:20.556466 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:20.556472 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:20.556494 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:20.556824 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:20.614836 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:20.615070 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:20.620877 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:20.619744 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:20.705253 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:20.740424 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:20.734282 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:20.760111 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:20.876068 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:20.887021 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:20.911174 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:20.910552 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0725 23:24:20.999224 281473590852288 failure_handler_test.py:195] epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0725 23:24:21.000774 281473590852288 failure_handler_test.py:195] epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0725 23:24:21.005640 281473590852288 failure_handler_test.py:195] epoch 0 finished [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0725 23:24:21.007612 281473590852288 failure_handler_test.py:195] epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:21.012514 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:21.027073 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:21.030468 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:21.028622 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:21.120474 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:21.110865 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:21.130807 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:21.120344 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:21.266682 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:21.240630 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:21.235937 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:21.261751 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:21.381059 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:21.412206 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:21.420405 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:21.448608 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:21.562253 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:21.568464 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:21.570237 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:21.570233 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:21.681615 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:21.681615 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:21.681319 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:21.710669 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending sigterm I0725 23:24:21.837541 281473054833344 failure_handler_test.py:302] sending sigterm INFO:tensorflow:sigterm sent I0725 23:24:21.837942 281473054833344 failure_handler_test.py:306] sigterm sent [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:21.850706 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:21.850719 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:21.881036 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Member 2 has received termination notice. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-2]: I0725 23:24:21.907960 281473590852288 failure_handling.py:710] Member 2 has received termination notice. [worker-2]: INFO:tensorflow:Termination caught in main thread on preempted worker [worker-2]: I0725 23:24:21.908515 281473590852288 failure_handling.py:1159] Termination caught in main thread on preempted worker [worker-2]: INFO:tensorflow:RUN_TO_CHECKPOINT set to 22 [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-1]: I0725 23:24:21.928935 281447319400928 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-0]: I0725 23:24:21.926622 281450104156640 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-2]: I0725 23:24:21.927323 281473590852288 failure_handling.py:1168] RUN_TO_CHECKPOINT set to 22 [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-2]: I0725 23:24:21.928601 281447881109984 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 0 received [worker-2]: I0725 23:24:21.929387 281473590852288 failure_handling.py:1177] Sigterm acknowledgement from replica 0 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 1 received [worker-2]: I0725 23:24:21.931274 281473590852288 failure_handling.py:1177] Sigterm acknowledgement from replica 1 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 2 received [worker-2]: I0725 23:24:21.932393 281473590852288 failure_handling.py:1177] Sigterm acknowledgement from replica 2 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 3 received [worker-2]: I0725 23:24:21.933465 281473590852288 failure_handling.py:1177] Sigterm acknowledgement from replica 3 received [worker-3]: I0725 23:24:21.926469 281450699682272 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:21.961563 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: I0725 23:24:22.066722 281473590852288 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: I0725 23:24:22.109263 281473590852288 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90gek698vz/tmpf2sf0o33/workertemp_3/fh_ckpt [worker-3]: I0725 23:24:22.133405 281473590852288 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90gek698vz/tmpf2sf0o33/workertemp_3/fh_ckpt [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: I0725 23:24:22.126278 281473590852288 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: I0725 23:24:22.136538 281473590852288 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-3]: I0725 23:24:22.164855 281473590852288 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: I0725 23:24:22.165122 281473590852288 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-1]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90gek698vz/tmpf2sf0o33/workertemp_1/fh_ckpt [worker-1]: I0725 23:24:22.178148 281473590852288 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90gek698vz/tmpf2sf0o33/workertemp_1/fh_ckpt [worker-1]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-1]: I0725 23:24:22.199963 281473590852288 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-1]: I0725 23:24:22.200218 281473590852288 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90gek698vz/tmpf2sf0o33/workertemp_2/fh_ckpt [worker-0]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90gek698vz/tmpf2sf0o33/fh_ckpt [worker-2]: I0725 23:24:22.202156 281473590852288 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90gek698vz/tmpf2sf0o33/workertemp_2/fh_ckpt [worker-0]: I0725 23:24:22.199629 281473590852288 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90gek698vz/tmpf2sf0o33/fh_ckpt [worker-0]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-0]: I0725 23:24:22.203884 281473590852288 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: I0725 23:24:22.204080 281473590852288 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-2]: I0725 23:24:22.205170 281473590852288 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: I0725 23:24:22.205370 281473590852288 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. INFO:tensorflow:restarting workers I0725 23:24:23.842380 281473054833344 failure_handler_test.py:309] restarting workers INFO:tensorflow:workers restarted I0725 23:24:23.916260 281473054833344 failure_handler_test.py:313] workers restarted [worker-0]: I0725 23:24:24.001411 281473590852288 multi_process_runner.py:840] Subprocess with PID 1374058 (worker, 0) is now being started. [worker-0]: I0725 23:24:24.001892 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:46359", "localhost:41815", "localhost:46379", "localhost:34407"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0725 23:24:24.100944 281473590852288 multi_process_runner.py:840] Subprocess with PID 1374061 (worker, 1) is now being started. [worker-3]: I0725 23:24:24.101078 281473590852288 multi_process_runner.py:840] Subprocess with PID 1374081 (worker, 3) is now being started. [worker-1]: I0725 23:24:24.101423 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:46359", "localhost:41815", "localhost:46379", "localhost:34407"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-3]: I0725 23:24:24.101508 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:46359", "localhost:41815", "localhost:46379", "localhost:34407"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: I0725 23:24:24.111618 281473590852288 multi_process_runner.py:840] Subprocess with PID 1374068 (worker, 2) is now being started. [worker-2]: I0725 23:24:24.112113 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:46359", "localhost:41815", "localhost:46379", "localhost:34407"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: 2023-07-25 23:24:24.150847: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:34407 [worker-0]: 2023-07-25 23:24:24.203092: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:46359 [worker-0]: 2023-07-25 23:24:24.203807: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 980392843734323138 [worker-3]: 2023-07-25 23:24:24.204370: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-2]: 2023-07-25 23:24:24.209180: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:46379 [worker-0]: 2023-07-25 23:24:24.212134: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 12850622397953394366 [worker-0]: 2023-07-25 23:24:24.212405: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-07-25 23:24:24.215447: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 12625846881677574568 [worker-2]: 2023-07-25 23:24:24.215915: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-1]: 2023-07-25 23:24:24.312437: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:41815 [worker-0]: 2023-07-25 23:24:24.338980: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 10771775553548695053 [worker-1]: 2023-07-25 23:24:24.339509: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0725 23:24:24.341438 281473590852288 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0725 23:24:24.347061 281473590852288 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0725 23:24:24.354945 281473590852288 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0725 23:24:24.357095 281473590852288 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0725 23:24:24.401362 281473590852288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0725 23:24:24.403258 281473590852288 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46359', 'localhost:41815', 'localhost:46379', 'localhost:34407']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0725 23:24:24.403513 281473590852288 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46359', 'localhost:41815', 'localhost:46379', 'localhost:34407']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0725 23:24:24.399337 281473590852288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0725 23:24:24.399892 281473590852288 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46359', 'localhost:41815', 'localhost:46379', 'localhost:34407']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0725 23:24:24.400127 281473590852288 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46359', 'localhost:41815', 'localhost:46379', 'localhost:34407']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0725 23:24:24.414165 281473590852288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0725 23:24:24.414749 281473590852288 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46359', 'localhost:41815', 'localhost:46379', 'localhost:34407']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0725 23:24:24.414986 281473590852288 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46359', 'localhost:41815', 'localhost:46379', 'localhost:34407']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0725 23:24:24.414095 281473590852288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0725 23:24:24.414832 281473590852288 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46359', 'localhost:41815', 'localhost:46379', 'localhost:34407']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0725 23:24:24.415073 281473590852288 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46359', 'localhost:41815', 'localhost:46379', 'localhost:34407']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0725 23:24:24.623549 281473590852288 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: I0725 23:24:24.623562 281473590852288 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0725 23:24:24.633336 281473590852288 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I0725 23:24:24.634668 281473590852288 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0725 23:24:24.635000 281473590852288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0725 23:24:24.635359 281473590852288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 22 [worker-0]: I0725 23:24:24.635551 281473590852288 failure_handler_test.py:197] Start training at 22 [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0725 23:24:24.643177 281473590852288 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I0725 23:24:24.644143 281473590852288 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0725 23:24:24.644418 281473590852288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: I0725 23:24:24.644152 281473590852288 failure_handling.py:674] Start watcher for local signal. [worker-2]: W0725 23:24:24.644747 281473590852288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0725 23:24:24.644567 281473590852288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: I0725 23:24:24.646518 281473590852288 failure_handling.py:674] Start watcher for local signal. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: Instructions for updating: [worker-3]: I0725 23:24:24.646921 281473590852288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: W0725 23:24:24.644919 281473590852288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-1]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Instructions for updating: [worker-1]: INFO:tensorflow:Start training at 22 [worker-3]: W0725 23:24:24.647270 281473590852288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: I0725 23:24:24.645114 281473590852288 failure_handler_test.py:197] Start training at 22 [worker-3]: Instructions for updating: [worker-2]: INFO:tensorflow:Start training at 22 [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: I0725 23:24:24.644937 281473590852288 failure_handler_test.py:197] Start training at 22 [worker-3]: INFO:tensorflow:Start training at 22 [worker-3]: I0725 23:24:24.647483 281473590852288 failure_handler_test.py:197] Start training at 22 [worker-2]: INFO:tensorflow:training restarted [worker-2]: I0725 23:24:24.678333 281473590852288 failure_handler_test.py:207] training restarted [worker-0]: INFO:tensorflow:training restarted [worker-0]: I0725 23:24:24.722073 281473590852288 failure_handler_test.py:207] training restarted [worker-3]: INFO:tensorflow:training restarted [worker-3]: I0725 23:24:24.737302 281473590852288 failure_handler_test.py:207] training restarted [worker-1]: INFO:tensorflow:training restarted [worker-1]: I0725 23:24:24.738454 281473590852288 failure_handler_test.py:207] training restarted [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:24.848694 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:24.904608 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:24.938840 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:24.947954 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:25.061653 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:25.071325 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:25.080999 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:25.097437 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:25.289206 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:25.301928 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:25.302052 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:25.302433 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:25.400851 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:25.401383 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:25.404015 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:25.409678 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:25.521821 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:25.536938 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:25.521821 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:25.583572 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffa8f98b80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0725 23:24:25.682452 281473590852288 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffa8f98b80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffa8f9ab60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0725 23:24:25.686342 281473590852288 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffa8f9ab60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:25.696469 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffa8f9a8e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0725 23:24:25.697088 281473590852288 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffa8f9a8e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffa8f98360> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0725 23:24:25.702238 281473590852288 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffa8f98360> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:25.722382 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:25.724602 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:25.757026 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9aac0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0725 23:24:25.839365 281473590852288 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9aac0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9b880> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0725 23:24:25.841001 281473590852288 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9b880> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9b2e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffa8f99d00> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0725 23:24:25.845568 281473590852288 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9b2e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0725 23:24:25.845650 281473590852288 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffa8f99d00> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:25.855112 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:25.855096 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:25.851426 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:25.855859 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:25.961936 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:25.963376 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:25.954304 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:25.985766 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0725 23:24:26.098109 281473590852288 failure_handler_test.py:195] epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0725 23:24:26.107126 281473590852288 failure_handler_test.py:195] epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-1]: I0725 23:24:26.108673 281473590852288 failure_handler_test.py:195] epoch 1 finished [worker-2]: I0725 23:24:26.108875 281473590852288 failure_handler_test.py:195] epoch 1 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:26.118924 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:26.120886 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:26.121113 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:26.181715 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:26.285184 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:26.290787 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:26.320786 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:26.320854 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:26.386415 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:26.396477 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:26.399713 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:26.407339 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:26.486716 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:26.487787 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:26.486738 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:26.486576 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:26.717803 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:26.720814 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:26.720480 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:26.720978 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:26.813035 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:26.813164 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:26.813241 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:26.813892 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:26.886897 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:26.892734 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:26.900100 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:26.898925 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:27.036802 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:27.038463 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:27.045094 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:27.049826 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:27.147016 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:27.161654 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:27.150761 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:27.170694 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:27.277884 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:27.291055 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:27.311315 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:27.327273 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:27.496147 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:27.527410 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:27.536261 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:27.536347 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:27.618496 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:27.640794 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:27.650662 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:27.651699 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:27.810556 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:27.815062 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:27.807620 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:27.819555 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:27.896704 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:27.921071 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:27.950677 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:27.930525 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:28.022223 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:28.022223 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:28.022396 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:28.085958 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-3]: INFO:tensorflow:epoch 2 finished [worker-0]: I0725 23:24:28.195386 281473590852288 failure_handler_test.py:195] epoch 2 finished [worker-3]: I0725 23:24:28.195404 281473590852288 failure_handler_test.py:195] epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0725 23:24:28.207270 281473590852288 failure_handler_test.py:195] epoch 2 finished [worker-0]: I0725 23:24:28.205885 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:28.205872 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0725 23:24:28.226492 281473590852288 failure_handler_test.py:195] epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:28.217093 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:28.236757 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:28.430721 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:28.420669 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:28.751203 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:29.445141 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:29.587170 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:29.610589 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:29.620966 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:29.651777 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:29.760791 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:29.760655 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:29.770656 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:29.785655 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:29.887197 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:29.895358 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:29.907034 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:29.914704 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:30.053620 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:30.061759 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:30.111531 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:30.111511 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:30.218123 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:30.246650 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:30.250385 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:30.261115 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:30.350548 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:30.347198 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:30.347306 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:30.347257 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:30.414449 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:30.415142 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:30.415441 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:30.414468 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:30.487556 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:30.489176 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:30.490380 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:30.488971 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:30.576199 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:30.576203 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:30.576059 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:30.576428 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:30.654368 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:30.654755 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:30.671454 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:30.686468 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:30.750406 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:30.767075 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:30.770943 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:30.774296 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:30.877041 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:30.882841 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:30.882897 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:30.900943 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:30.981194 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:30.985872 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:30.990325 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:30.985883 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0725 23:24:31.079828 281473590852288 failure_handler_test.py:195] epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:31.090219 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-0]: I0725 23:24:31.101690 281473590852288 failure_handler_test.py:195] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-1]: I0725 23:24:31.101894 281473590852288 failure_handler_test.py:195] epoch 3 finished [worker-2]: I0725 23:24:31.102273 281473590852288 failure_handler_test.py:195] epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:31.111116 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:31.111932 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:31.112403 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:31.237199 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:31.251789 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:31.256519 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:31.268741 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:31.339488 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:31.340526 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:31.340596 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:31.340787 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:31.490708 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:31.500922 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:31.481453 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:31.521642 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:31.631773 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:31.661516 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:31.671165 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:31.671134 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:31.920446 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:31.931201 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:31.955208 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:31.957045 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:32.227494 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:32.235145 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:32.232280 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:32.253047 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:32.312299 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:32.312355 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:32.312310 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:32.312330 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:32.409136 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:32.406789 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:32.406787 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:32.415860 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:32.479336 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:32.480605 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:32.489972 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:32.485686 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:32.588331 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:32.589573 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:32.604962 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:32.592264 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:32.676460 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:32.686249 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:32.686371 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:32.690470 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:32.771581 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:32.783673 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:32.791022 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:32.790909 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:32.866292 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:32.866339 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:32.871082 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:32.871118 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:32.967708 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:32.964241 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:32.964240 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:32.963550 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-3]: I0725 23:24:33.047734 281473590852288 failure_handler_test.py:195] epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-1]: I0725 23:24:33.048147 281473590852288 failure_handler_test.py:195] epoch 4 finished [worker-2]: INFO:tensorflow:epoch 4 finished [worker-0]: I0725 23:24:33.048079 281473590852288 failure_handler_test.py:195] epoch 4 finished [worker-2]: I0725 23:24:33.048377 281473590852288 failure_handler_test.py:195] epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:33.057719 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:33.057964 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:33.077003 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:33.080543 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:33.183487 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:33.176884 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:33.200903 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:33.200904 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:33.363538 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:33.391142 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:33.407369 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:33.391216 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:33.493063 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:33.492737 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:33.500949 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:33.496027 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:33.559854 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:33.560061 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:33.560582 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:33.560982 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:33.645927 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:33.647097 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:33.649616 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:33.650713 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:33.713880 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:33.714669 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:33.714704 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:33.714707 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:33.775548 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:33.775868 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:33.777167 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:33.776786 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:33.836623 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:33.844969 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:33.846389 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:33.837345 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:33.929453 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:33.931826 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:33.932823 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:33.936596 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:33.996980 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:33.997032 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:33.997077 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:33.999695 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:34.092996 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:34.110740 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:34.110751 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:34.114947 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:34.236298 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:34.271485 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:34.261334 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:34.270836 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:34.416095 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:34.446246 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:34.471211 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:34.482068 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:34.621513 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:34.630916 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:34.630916 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:34.686924 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 5 finished [worker-3]: I0725 23:24:34.738357 281473590852288 failure_handler_test.py:195] epoch 5 finished [worker-0]: INFO:tensorflow:epoch 5 finished [worker-1]: INFO:tensorflow:epoch 5 finished [worker-1]: I0725 23:24:34.748717 281473590852288 failure_handler_test.py:195] epoch 5 finished [worker-0]: I0725 23:24:34.747149 281473590852288 failure_handler_test.py:195] epoch 5 finished [worker-2]: INFO:tensorflow:epoch 5 finished [worker-2]: I0725 23:24:34.755284 281473590852288 failure_handler_test.py:195] epoch 5 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:34.771804 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:34.781747 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:34.790257 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:34.791576 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:34.961110 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:34.961040 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:34.990819 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:34.996991 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:35.097048 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:35.108763 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:35.124515 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:35.120645 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:35.223760 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:35.225390 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:35.241318 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:35.250192 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:35.310889 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:35.312006 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:35.316293 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:35.327247 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:35.389378 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:35.389598 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:35.389813 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:35.390333 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:35.457724 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:35.458417 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:35.458303 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:35.458281 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:35.519223 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:35.519285 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:35.542121 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:35.546272 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:35.642169 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:35.648082 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:35.648782 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:35.649277 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:35.727468 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:35.728183 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:35.727937 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:35.727846 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:35.824213 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:35.830728 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:35.840893 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:35.840866 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:36.016503 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:36.016554 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:36.011283 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:36.043047 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:36.104738 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:36.104876 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:36.105496 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:36.105799 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:36.174455 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:36.174489 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:36.174931 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:36.177649 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:36.238047 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:36.238232 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:36.238533 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:36.240665 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 6 finished [worker-0]: INFO:tensorflow:epoch 6 finished [worker-2]: INFO:tensorflow:epoch 6 finished [worker-3]: I0725 23:24:36.298119 281473590852288 failure_handler_test.py:195] epoch 6 finished [worker-1]: INFO:tensorflow:epoch 6 finished [worker-0]: I0725 23:24:36.298547 281473590852288 failure_handler_test.py:195] epoch 6 finished [worker-2]: I0725 23:24:36.298686 281473590852288 failure_handler_test.py:195] epoch 6 finished [worker-1]: I0725 23:24:36.298625 281473590852288 failure_handler_test.py:195] epoch 6 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:36.308162 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:36.308310 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:36.308288 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:36.308257 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:36.421025 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:36.428339 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:36.433554 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:36.470901 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:36.549911 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:36.562235 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:36.571281 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:36.581275 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:36.641417 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:36.641316 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:36.641265 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:36.641301 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:36.708549 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:36.708592 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:36.708705 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:36.725631 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:36.785974 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:36.786828 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:36.787145 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:36.791211 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:36.856864 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:36.857034 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:36.858836 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:36.868143 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:36.926260 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:36.957032 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:36.959602 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:36.964162 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:37.028069 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:37.031890 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:37.041065 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:37.057130 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:37.150432 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:37.150432 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:37.153233 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:37.160985 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:37.223887 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:37.223898 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:37.222898 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:37.235180 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:37.297467 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:37.299225 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:37.300822 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:37.301766 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:37.378405 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:37.381924 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:37.384313 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:37.384310 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:37.467853 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:37.466706 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:37.480684 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:37.486467 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:37.546156 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:37.545616 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:37.551482 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:37.557231 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 7 finished [worker-3]: I0725 23:24:37.633283 281473590852288 failure_handler_test.py:195] epoch 7 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0725 23:24:37.634299 281473590852288 failure_handler_test.py:245] Training finished. [worker-1]: INFO:tensorflow:epoch 7 finished [worker-1]: I0725 23:24:37.637147 281473590852288 failure_handler_test.py:195] epoch 7 finished [worker-2]: INFO:tensorflow:epoch 7 finished [worker-1]: INFO:tensorflow:Training finished. [worker-2]: I0725 23:24:37.637473 281473590852288 failure_handler_test.py:195] epoch 7 finished [worker-1]: I0725 23:24:37.637981 281473590852288 failure_handler_test.py:245] Training finished. [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I0725 23:24:37.638564 281473590852288 failure_handler_test.py:245] Training finished. [worker-0]: INFO:tensorflow:epoch 7 finished [worker-0]: I0725 23:24:37.646629 281473590852288 failure_handler_test.py:195] epoch 7 finished [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I0725 23:24:37.648404 281473590852288 failure_handler_test.py:245] Training finished. I0725 23:24:37.929702 281473054833344 multi_process_runner.py:646] worker-0 exit code: 0 I0725 23:24:37.929962 281473054833344 multi_process_runner.py:646] worker-1 exit code: 0 I0725 23:24:37.930109 281473054833344 multi_process_runner.py:646] worker-2 exit code: 0 I0725 23:24:37.930248 281473054833344 multi_process_runner.py:646] worker-3 exit code: 0 I0725 23:24:37.932317 281473054833344 multi_process_runner.py:662] Joining log reading threads. I0725 23:24:37.932554 281473054833344 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker): 21.56s I0725 23:24:38.195262 281473054833344 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker): 21.56s [ OK ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 33027 I0725 23:24:38.197039 281473054833344 test_util.py:3813] Using local port 33027 INFO:tensorflow:Using local port 33751 I0725 23:24:38.197449 281473054833344 test_util.py:3813] Using local port 33751 INFO:tensorflow:Using local port 35721 I0725 23:24:38.197797 281473054833344 test_util.py:3813] Using local port 35721 INFO:tensorflow:Using local port 37749 I0725 23:24:38.198136 281473054833344 test_util.py:3813] Using local port 37749 INFO:tensorflow:Cluster starting. I0725 23:24:38.229085 281473054833344 failure_handler_test.py:297] Cluster starting. [worker-1]: I0725 23:24:38.363564 281473590852288 multi_process_runner.py:840] Subprocess with PID 1386893 (worker, 1) is now being started. [worker-1]: I0725 23:24:38.364013 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:33027", "localhost:33751", "localhost:35721", "localhost:37749"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: I0725 23:24:38.377317 281473590852288 multi_process_runner.py:840] Subprocess with PID 1386816 (worker, 0) is now being started. [worker-3]: I0725 23:24:38.379509 281473590852288 multi_process_runner.py:840] Subprocess with PID 1387040 (worker, 3) is now being started. [worker-2]: I0725 23:24:38.407018 281473590852288 multi_process_runner.py:840] Subprocess with PID 1386902 (worker, 2) is now being started. [worker-0]: I0725 23:24:38.377779 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:33027", "localhost:33751", "localhost:35721", "localhost:37749"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-3]: I0725 23:24:38.379954 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:33027", "localhost:33751", "localhost:35721", "localhost:37749"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: I0725 23:24:38.407467 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:33027", "localhost:33751", "localhost:35721", "localhost:37749"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: 2023-07-25 23:24:38.455478: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:37749 [worker-1]: 2023-07-25 23:24:38.467058: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:33751 [worker-2]: 2023-07-25 23:24:38.487748: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:35721 [worker-0]: 2023-07-25 23:24:38.596853: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:33027 [worker-0]: 2023-07-25 23:24:38.598320: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 13467460097998004002 [worker-1]: 2023-07-25 23:24:38.598642: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-07-25 23:24:38.605024: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 18431723106499109262 [worker-2]: 2023-07-25 23:24:38.605246: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-07-25 23:24:38.619660: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 14363571491785679435 [worker-3]: 2023-07-25 23:24:38.619958: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-07-25 23:24:38.639637: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 9244648416406781051 [worker-0]: 2023-07-25 23:24:38.639949: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0725 23:24:38.642147 281473590852288 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: I0725 23:24:38.642200 281473590852288 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0725 23:24:38.647387 281473590852288 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0725 23:24:38.642422 281473590852288 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0725 23:24:38.698603 281473590852288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0725 23:24:38.699209 281473590852288 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33027', 'localhost:33751', 'localhost:35721', 'localhost:37749']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0725 23:24:38.699449 281473590852288 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33027', 'localhost:33751', 'localhost:35721', 'localhost:37749']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0725 23:24:38.700822 281473590852288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0725 23:24:38.701426 281473590852288 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33027', 'localhost:33751', 'localhost:35721', 'localhost:37749']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0725 23:24:38.701670 281473590852288 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33027', 'localhost:33751', 'localhost:35721', 'localhost:37749']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-3]: I0725 23:24:38.721390 281473590852288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-1]: I0725 23:24:38.721390 281473590852288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-1]: INFO:tensorflow:Check health not enabled. [worker-3]: I0725 23:24:38.722034 281473590852288 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: I0725 23:24:38.722039 281473590852288 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33027', 'localhost:33751', 'localhost:35721', 'localhost:37749']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33027', 'localhost:33751', 'localhost:35721', 'localhost:37749']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0725 23:24:38.722274 281473590852288 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33027', 'localhost:33751', 'localhost:35721', 'localhost:37749']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0725 23:24:38.722279 281473590852288 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33027', 'localhost:33751', 'localhost:35721', 'localhost:37749']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0725 23:24:38.764989 281473590852288 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: I0725 23:24:38.765018 281473590852288 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I0725 23:24:38.764775 281473590852288 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: I0725 23:24:38.765089 281473590852288 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: I0725 23:24:38.765758 281473590852288 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0725 23:24:38.767646 281473590852288 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0725 23:24:38.766023 281473590852288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: I0725 23:24:38.768007 281473590852288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-3]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0725 23:24:38.766355 281473590852288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: W0725 23:24:38.768345 281473590852288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-3]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: I0725 23:24:38.766547 281473590852288 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0725 23:24:38.768542 281473590852288 failure_handler_test.py:197] Start training at 0 [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I0725 23:24:38.776521 281473590852288 failure_handling.py:674] Start watcher for local signal. [worker-1]: I0725 23:24:38.776588 281473590852288 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0725 23:24:38.776961 281473590852288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: I0725 23:24:38.777016 281473590852288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-1]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0725 23:24:38.777310 281473590852288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: W0725 23:24:38.777359 281473590852288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-1]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-1]: INFO:tensorflow:Start training at 0 [worker-0]: I0725 23:24:38.777507 281473590852288 failure_handler_test.py:197] Start training at 0 [worker-1]: I0725 23:24:38.777555 281473590852288 failure_handler_test.py:197] Start training at 0 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:38.946322 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:38.975640 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:38.975785 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:39.012642 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:39.084595 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:39.088795 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:39.090412 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:39.090768 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:39.170425 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:39.175843 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:39.190329 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:39.213493 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:39.286734 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:39.289366 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:39.291662 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:39.301495 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:39.364566 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:39.364613 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:39.364624 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:39.365301 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffa8f90860> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffa8f92660> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffa8f9a980> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffa8f96840> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0725 23:24:39.415669 281473590852288 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffa8f90860> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0725 23:24:39.416072 281473590852288 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffa8f96840> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0725 23:24:39.415825 281473590852288 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffa8f92660> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0725 23:24:39.415992 281473590852288 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffa8f9a980> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:39.425238 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:39.425290 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:39.437201 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:39.437200 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffa8f90720> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffa8f90fe0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffa8f96480> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9b880> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0725 23:24:39.498690 281473590852288 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffa8f90720> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0725 23:24:39.498982 281473590852288 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffa8f96480> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0725 23:24:39.498788 281473590852288 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffa8f90fe0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0725 23:24:39.498965 281473590852288 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9b880> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:39.508064 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:39.508157 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:39.508092 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:39.508108 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:39.567103 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:39.567094 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:39.567794 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:39.567845 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:39.625692 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:39.625789 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:39.627222 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:39.627300 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:39.685529 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:39.685634 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:39.686766 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:39.687385 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:39.742524 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:39.742567 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:39.743238 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:39.743856 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:39.906574 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:39.907360 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:39.907891 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:39.908628 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:40.104843 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:40.105046 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:40.105114 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:40.112337 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:40.187165 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:40.186254 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:40.202596 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:40.220912 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:40.279114 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:40.279100 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:40.279117 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:40.279232 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 0 finished [worker-0]: INFO:tensorflow:epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-3]: I0725 23:24:40.331022 281473590852288 failure_handler_test.py:195] epoch 0 finished [worker-0]: I0725 23:24:40.331184 281473590852288 failure_handler_test.py:195] epoch 0 finished [worker-1]: I0725 23:24:40.331393 281473590852288 failure_handler_test.py:195] epoch 0 finished [worker-2]: I0725 23:24:40.331503 281473590852288 failure_handler_test.py:195] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:40.340905 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:40.341642 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:40.344849 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:40.342097 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:40.399926 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:40.399961 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:40.400122 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:40.401190 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:40.458068 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:40.461008 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:40.461600 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:40.458296 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:40.516479 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:40.516587 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:40.516590 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:40.520176 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:40.584556 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:40.585016 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:40.587687 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:40.586042 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:40.645673 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:40.646761 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:40.644681 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:40.649365 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:40.711792 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:40.713126 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:40.730838 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:40.730965 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:40.827026 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:40.831135 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:40.826364 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:40.845592 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:40.914902 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:40.913539 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:40.917720 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:40.913496 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:40.979072 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:40.980160 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:40.979152 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:40.996635 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:41.118185 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:41.105385 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:41.125352 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:41.127027 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:41.185586 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:41.185521 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:41.185752 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:41.185915 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:41.247934 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:41.248481 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:41.248991 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:41.249058 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending sigterm I0725 23:24:41.289489 281473054833344 failure_handler_test.py:302] sending sigterm INFO:tensorflow:sigterm sent I0725 23:24:41.289833 281473054833344 failure_handler_test.py:306] sigterm sent [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:41.324264 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Member 2 has received termination notice. [worker-2]: I0725 23:24:41.329126 281473590852288 failure_handling.py:710] Member 2 has received termination notice. [worker-2]: INFO:tensorflow:Termination caught in main thread on preempted worker [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:41.329737 281473590852288 failure_handling.py:1159] Termination caught in main thread on preempted worker [worker-0]: I0725 23:24:41.329971 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:RUN_TO_CHECKPOINT set to 29 [worker-2]: I0725 23:24:41.330889 281473590852288 failure_handling.py:1168] RUN_TO_CHECKPOINT set to 29 [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-2]: I0725 23:24:41.331025 281447705145824 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 0 received [worker-2]: I0725 23:24:41.332820 281473590852288 failure_handling.py:1177] Sigterm acknowledgement from replica 0 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 1 received [worker-2]: I0725 23:24:41.333391 281473590852288 failure_handling.py:1177] Sigterm acknowledgement from replica 1 received [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 2 received [worker-1]: I0725 23:24:41.332708 281448845996512 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-2]: I0725 23:24:41.333880 281473590852288 failure_handling.py:1177] Sigterm acknowledgement from replica 2 received [worker-0]: I0725 23:24:41.333739 281447856271840 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:41.335916 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 3 received [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-3]: I0725 23:24:41.352638 281447973581280 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-2]: I0725 23:24:41.346869 281473590852288 failure_handling.py:1177] Sigterm acknowledgement from replica 3 received [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:41.356130 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: I0725 23:24:41.405117 281473590852288 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: I0725 23:24:41.416697 281473590852288 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: I0725 23:24:41.416698 281473590852288 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: I0725 23:24:41.417065 281473590852288 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90gek698vz/tmprm7kak7h/workertemp_3/fh_ckpt [worker-3]: I0725 23:24:41.450973 281473590852288 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90gek698vz/tmprm7kak7h/workertemp_3/fh_ckpt [worker-3]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-3]: I0725 23:24:41.452480 281473590852288 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: I0725 23:24:41.452711 281473590852288 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90gek698vz/tmprm7kak7h/fh_ckpt [worker-1]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90gek698vz/tmprm7kak7h/workertemp_1/fh_ckpt [worker-1]: I0725 23:24:41.462136 281473590852288 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90gek698vz/tmprm7kak7h/workertemp_1/fh_ckpt [worker-2]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90gek698vz/tmprm7kak7h/workertemp_2/fh_ckpt [worker-2]: I0725 23:24:41.462502 281473590852288 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90gek698vz/tmprm7kak7h/workertemp_2/fh_ckpt [worker-1]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-2]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-1]: I0725 23:24:41.464029 281473590852288 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: I0725 23:24:41.464284 281473590852288 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-1]: I0725 23:24:41.464240 281473590852288 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: I0725 23:24:41.462063 281473590852288 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90gek698vz/tmprm7kak7h/fh_ckpt [worker-2]: I0725 23:24:41.464492 281473590852288 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-0]: I0725 23:24:41.463882 281473590852288 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: I0725 23:24:41.464081 281473590852288 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. INFO:tensorflow:restarting workers I0725 23:24:43.290495 281473054833344 failure_handler_test.py:309] restarting workers [worker-0]: I0725 23:24:43.337003 281473590852288 multi_process_runner.py:840] Subprocess with PID 1406511 (worker, 0) is now being started. [worker-0]: I0725 23:24:43.337396 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:33027", "localhost:33751", "localhost:35721", "localhost:37749"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' INFO:tensorflow:workers restarted I0725 23:24:43.363783 281473054833344 failure_handler_test.py:313] workers restarted [worker-0]: 2023-07-25 23:24:43.383011: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:33027 [worker-2]: I0725 23:24:43.425512 281473590852288 multi_process_runner.py:840] Subprocess with PID 1406730 (worker, 2) is now being started. [worker-3]: I0725 23:24:43.425575 281473590852288 multi_process_runner.py:840] Subprocess with PID 1406918 (worker, 3) is now being started. [worker-1]: I0725 23:24:43.426591 281473590852288 multi_process_runner.py:840] Subprocess with PID 1406614 (worker, 1) is now being started. [worker-0]: 2023-07-25 23:24:43.427372: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 2013410253822770140 [worker-0]: 2023-07-25 23:24:43.427853: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: I0725 23:24:43.425938 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:33027", "localhost:33751", "localhost:35721", "localhost:37749"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-1]: I0725 23:24:43.426954 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:33027", "localhost:33751", "localhost:35721", "localhost:37749"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0725 23:24:43.425923 281473590852288 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:33027", "localhost:33751", "localhost:35721", "localhost:37749"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-2]: 2023-07-25 23:24:43.466606: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:35721 [worker-0]: 2023-07-25 23:24:43.532623: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 4056505103522689044 [worker-2]: 2023-07-25 23:24:43.532855: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: 2023-07-25 23:24:43.539028: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:37749 [worker-0]: 2023-07-25 23:24:43.552371: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 5937621790882039538 [worker-3]: 2023-07-25 23:24:43.553115: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-1]: 2023-07-25 23:24:43.582059: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:33751 [worker-0]: 2023-07-25 23:24:43.607593: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 7951998898076901476 [worker-1]: 2023-07-25 23:24:43.610302: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0725 23:24:43.613185 281473590852288 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0725 23:24:43.612790 281473590852288 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0725 23:24:43.612775 281473590852288 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0725 23:24:43.628821 281473590852288 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0725 23:24:43.671420 281473590852288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0725 23:24:43.671982 281473590852288 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33027', 'localhost:33751', 'localhost:35721', 'localhost:37749']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0725 23:24:43.672211 281473590852288 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33027', 'localhost:33751', 'localhost:35721', 'localhost:37749']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0725 23:24:43.703864 281473590852288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0725 23:24:43.704405 281473590852288 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33027', 'localhost:33751', 'localhost:35721', 'localhost:37749']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0725 23:24:43.704635 281473590852288 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33027', 'localhost:33751', 'localhost:35721', 'localhost:37749']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0725 23:24:43.721767 281473590852288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0725 23:24:43.722265 281473590852288 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33027', 'localhost:33751', 'localhost:35721', 'localhost:37749']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0725 23:24:43.722496 281473590852288 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33027', 'localhost:33751', 'localhost:35721', 'localhost:37749']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0725 23:24:43.735494 281473590852288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0725 23:24:43.736515 281473590852288 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33027', 'localhost:33751', 'localhost:35721', 'localhost:37749']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0725 23:24:43.736751 281473590852288 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33027', 'localhost:33751', 'localhost:35721', 'localhost:37749']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0725 23:24:43.812695 281473590852288 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I0725 23:24:43.813462 281473590852288 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0725 23:24:43.813877 281473590852288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0725 23:24:43.814235 281473590852288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 29 [worker-0]: I0725 23:24:43.814430 281473590852288 failure_handler_test.py:197] Start training at 29 [worker-0]: INFO:tensorflow:training restarted [worker-0]: I0725 23:24:43.822759 281473590852288 failure_handler_test.py:207] training restarted [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0725 23:24:43.812660 281473590852288 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I0725 23:24:43.813577 281473590852288 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0725 23:24:43.813839 281473590852288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0725 23:24:43.814145 281473590852288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 29 [worker-2]: I0725 23:24:43.814863 281473590852288 failure_handler_test.py:197] Start training at 29 [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0725 23:24:43.832643 281473590852288 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: I0725 23:24:43.832657 281473590852288 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:training restarted [worker-2]: I0725 23:24:43.840853 281473590852288 failure_handler_test.py:207] training restarted [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I0725 23:24:43.847506 281473590852288 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0725 23:24:43.847860 281473590852288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0725 23:24:43.848171 281473590852288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 29 [worker-3]: I0725 23:24:43.848355 281473590852288 failure_handler_test.py:197] Start training at 29 [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0725 23:24:43.856548 281473590852288 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0725 23:24:43.856922 281473590852288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0725 23:24:43.857228 281473590852288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 29 [worker-1]: I0725 23:24:43.857411 281473590852288 failure_handler_test.py:197] Start training at 29 [worker-1]: INFO:tensorflow:training restarted [worker-1]: I0725 23:24:43.877336 281473590852288 failure_handler_test.py:207] training restarted [worker-3]: INFO:tensorflow:training restarted [worker-3]: I0725 23:24:43.901734 281473590852288 failure_handler_test.py:207] training restarted [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:43.991310 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:44.054645 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:44.076655 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:44.083830 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-3]: I0725 23:24:44.142360 281473590852288 failure_handler_test.py:195] epoch 1 finished [worker-1]: I0725 23:24:44.142550 281473590852288 failure_handler_test.py:195] epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-2]: I0725 23:24:44.142604 281473590852288 failure_handler_test.py:195] epoch 1 finished [worker-0]: I0725 23:24:44.142587 281473590852288 failure_handler_test.py:195] epoch 1 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:44.153406 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:44.153546 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:44.152980 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:44.171148 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:44.285760 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:44.285633 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:44.301538 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:44.313084 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:44.382933 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:44.382932 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:44.389496 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:44.427643 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:44.511348 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:44.512717 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:44.524308 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:44.541222 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffa8f98a40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0725 23:24:44.616208 281473590852288 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffa8f98a40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffa8f9e020> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0725 23:24:44.628681 281473590852288 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffa8f9e020> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:44.625039 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:44.638661 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffa8f9c540> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0725 23:24:44.652828 281473590852288 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffa8f9c540> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffa8f99080> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0725 23:24:44.652543 281473590852288 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffa8f99080> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:44.663120 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:44.664301 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9ad40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9c040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9bb00> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9f920> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0725 23:24:44.721393 281473590852288 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9f920> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0725 23:24:44.720991 281473590852288 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9bb00> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0725 23:24:44.721129 281473590852288 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9c040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0725 23:24:44.720787 281473590852288 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffa8f9ad40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:44.729780 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:44.729997 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:44.732592 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:44.729893 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:44.798738 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:44.798811 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:44.799149 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:44.800029 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:44.861459 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:44.861491 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:44.861266 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:44.876859 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:44.962469 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:44.964595 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:44.980694 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:45.010876 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:45.080780 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:45.081664 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:45.080510 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:45.101383 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:45.191425 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:45.192522 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:45.195380 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:45.191324 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:45.434435 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:45.434602 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:45.434514 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:45.434467 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:45.514558 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:45.514557 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:45.514863 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:45.514675 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:45.574050 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:45.574552 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:45.574380 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:45.574741 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:45.650495 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:45.675215 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:45.675441 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:45.675803 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0725 23:24:45.730146 281473590852288 failure_handler_test.py:195] epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I0725 23:24:45.732566 281473590852288 failure_handler_test.py:195] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0725 23:24:45.732683 281473590852288 failure_handler_test.py:195] epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0725 23:24:45.743541 281473590852288 failure_handler_test.py:195] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:45.748230 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:45.748238 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:45.741479 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:45.790430 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:45.851605 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:45.851806 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:45.853750 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:45.853854 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:45.946010 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:45.965433 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:45.987049 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:45.996783 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:46.107255 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:46.111306 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:46.110563 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:46.110702 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:46.244273 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:46.245308 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:46.254235 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:46.261971 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:46.324383 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:46.323898 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:46.330902 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:46.327199 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:46.408193 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:46.409871 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:46.408478 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:46.450976 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:46.510468 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:46.512239 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:46.523014 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:46.515142 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:46.592091 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:46.591216 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:46.589974 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:46.607028 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:46.689233 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:46.688026 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:46.697340 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:46.696863 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:46.796310 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:46.796217 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:46.795792 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:46.840546 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:46.932084 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:46.930384 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:46.942151 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:46.960686 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:47.130435 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:47.119329 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:47.134148 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:47.109788 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:47.236003 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:47.244841 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:47.243191 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:47.260780 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:47.386643 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:47.396046 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:47.398693 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:47.396662 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-3]: I0725 23:24:47.481363 281473590852288 failure_handler_test.py:195] epoch 3 finished [worker-1]: I0725 23:24:47.481733 281473590852288 failure_handler_test.py:195] epoch 3 finished [worker-0]: I0725 23:24:47.481536 281473590852288 failure_handler_test.py:195] epoch 3 finished [worker-2]: I0725 23:24:47.481833 281473590852288 failure_handler_test.py:195] epoch 3 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:47.490914 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:47.490913 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:47.510585 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:47.527066 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:47.666964 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:47.696057 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:47.700234 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:47.710366 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:47.788264 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:47.810809 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:47.810360 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:47.841918 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:47.944656 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:47.940326 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:47.955636 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:47.940295 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:48.030886 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:48.036073 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:48.051279 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:48.066801 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:48.166783 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:48.164444 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:48.164713 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:48.188033 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:48.277524 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:48.279328 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:48.282045 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:48.320638 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:48.384191 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:48.390388 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:48.395938 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:48.387400 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:48.475290 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:48.485960 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:48.481052 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:48.485461 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:48.557991 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:48.558051 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:48.563721 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:48.570662 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:48.665369 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:48.675457 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:48.675710 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:48.681247 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:48.745735 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:48.746514 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:48.760706 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:48.780652 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:48.907444 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:48.920059 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:48.923848 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:48.930668 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:49.001337 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:49.003663 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:49.008079 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:49.026037 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:49.126558 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:49.137811 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:49.141348 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:49.149837 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0725 23:24:49.243135 281473590852288 failure_handler_test.py:195] epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I0725 23:24:49.247370 281473590852288 failure_handler_test.py:195] epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:49.252454 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-2]: I0725 23:24:49.248701 281473590852288 failure_handler_test.py:195] epoch 4 finished [worker-0]: I0725 23:24:49.254379 281473590852288 failure_handler_test.py:195] epoch 4 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:49.258081 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:49.263332 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:49.280841 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:49.339799 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:49.341812 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:49.343511 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:49.339926 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:49.415983 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:49.415954 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:49.416082 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:49.452654 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:49.533646 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:49.534511 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:49.558530 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:49.550555 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:49.665454 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:49.684315 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:49.687216 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:49.690505 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:49.756955 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:49.761446 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:49.780788 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:49.780779 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:49.865685 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:49.865843 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:49.882970 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:49.879731 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:49.951398 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:49.965021 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:49.968654 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:49.980618 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:50.087716 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:50.087599 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:50.087584 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:50.110365 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:50.276319 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:50.292571 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:50.296275 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:50.292546 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:50.389337 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:50.395541 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:50.419760 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:50.395085 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:50.493540 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:50.500545 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:50.516152 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:50.530877 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:50.627724 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:50.629341 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:50.639303 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:50.644865 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:50.764245 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:50.772913 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:50.760737 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:50.767079 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:50.945788 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:50.948039 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:50.950880 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:50.957658 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 5 finished [worker-0]: INFO:tensorflow:epoch 5 finished [worker-0]: I0725 23:24:51.057163 281473590852288 failure_handler_test.py:195] epoch 5 finished [worker-3]: I0725 23:24:51.056928 281473590852288 failure_handler_test.py:195] epoch 5 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:51.066254 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:51.069941 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 5 finished [worker-1]: I0725 23:24:51.068824 281473590852288 failure_handler_test.py:195] epoch 5 finished [worker-2]: INFO:tensorflow:epoch 5 finished [worker-2]: I0725 23:24:51.076787 281473590852288 failure_handler_test.py:195] epoch 5 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:51.094939 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:51.077513 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:51.158108 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:51.158283 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:51.160778 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:51.158246 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:51.222873 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:51.225666 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:51.223012 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:51.222740 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:51.293521 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:51.294926 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:51.320560 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:51.320479 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:51.418828 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:51.423323 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:51.458607 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:51.470652 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:51.536517 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:51.536554 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:51.546292 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:51.547050 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:51.649729 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:51.641003 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:51.647204 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:51.672146 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:51.732222 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:51.730858 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:51.737854 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:51.731270 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:51.794833 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:51.794947 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:51.794151 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:51.794152 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:51.865925 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:51.865991 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:51.868288 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:51.877237 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:51.970263 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:51.979519 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:51.979831 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:51.980027 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:52.039891 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:52.039887 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:52.039927 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:52.040078 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:52.105742 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:52.112956 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:52.125725 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:52.149259 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:52.262156 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:52.270186 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:52.272208 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:52.276464 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:52.614845 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:52.605203 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:52.641701 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:52.662263 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 6 finished [worker-0]: I0725 23:24:52.798666 281473590852288 failure_handler_test.py:195] epoch 6 finished [worker-1]: INFO:tensorflow:epoch 6 finished [worker-1]: I0725 23:24:52.799796 281473590852288 failure_handler_test.py:195] epoch 6 finished [worker-2]: INFO:tensorflow:epoch 6 finished [worker-2]: I0725 23:24:52.799947 281473590852288 failure_handler_test.py:195] epoch 6 finished [worker-3]: INFO:tensorflow:epoch 6 finished [worker-3]: I0725 23:24:52.800447 281473590852288 failure_handler_test.py:195] epoch 6 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:52.809062 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:52.809502 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:52.809757 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:52.807863 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:52.973383 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:52.965265 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:52.990474 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:52.995841 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:53.113548 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:53.126704 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:53.150474 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:53.121482 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:53.222117 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:53.222396 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:53.231876 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:53.240395 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:53.410709 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:53.418765 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:53.418235 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:53.443328 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:53.518125 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:53.518125 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:53.518175 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:53.520597 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:53.599516 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:53.599615 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:53.600105 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:53.604507 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:53.816670 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:53.831497 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:53.821419 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:53.840404 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:53.977145 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:53.977719 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:53.982049 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:53.986791 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:54.055516 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:54.055526 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:54.056781 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:54.053878 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:54.149968 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:54.152138 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:54.153993 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:54.167348 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:54.250790 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:54.253345 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:54.271059 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:54.257730 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:54.387077 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:54.387141 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:54.393651 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:54.404802 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:54.468140 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:54.467889 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:54.468396 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:54.487410 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0725 23:24:54.572562 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0725 23:24:54.576531 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0725 23:24:54.569246 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0725 23:24:54.589961 281473590852288 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 7 finished [worker-1]: INFO:tensorflow:epoch 7 finished [worker-0]: INFO:tensorflow:epoch 7 finished [worker-3]: I0725 23:24:54.644680 281473590852288 failure_handler_test.py:195] epoch 7 finished [worker-2]: INFO:tensorflow:epoch 7 finished [worker-1]: I0725 23:24:54.645008 281473590852288 failure_handler_test.py:195] epoch 7 finished [worker-2]: I0725 23:24:54.645005 281473590852288 failure_handler_test.py:195] epoch 7 finished [worker-0]: I0725 23:24:54.644858 281473590852288 failure_handler_test.py:195] epoch 7 finished [worker-3]: INFO:tensorflow:Training finished. [worker-2]: INFO:tensorflow:Training finished. [worker-3]: I0725 23:24:54.646599 281473590852288 failure_handler_test.py:245] Training finished. [worker-1]: INFO:tensorflow:Training finished. [worker-2]: I0725 23:24:54.646215 281473590852288 failure_handler_test.py:245] Training finished. [worker-1]: I0725 23:24:54.647356 281473590852288 failure_handler_test.py:245] Training finished. [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I0725 23:24:54.651348 281473590852288 failure_handler_test.py:245] Training finished. I0725 23:24:55.357201 281473054833344 multi_process_runner.py:646] worker-0 exit code: 0 I0725 23:24:55.357461 281473054833344 multi_process_runner.py:646] worker-1 exit code: 0 I0725 23:24:55.357605 281473054833344 multi_process_runner.py:646] worker-2 exit code: 0 I0725 23:24:55.357739 281473054833344 multi_process_runner.py:646] worker-3 exit code: 0 I0725 23:24:55.359883 281473054833344 multi_process_runner.py:662] Joining log reading threads. I0725 23:24:55.360109 281473054833344 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker): 17.48s I0725 23:24:55.679979 281473054833344 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker): 17.48s [ OK ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker ====================================================================== ERROR: test_error_propagation (__main__.PreemptionCheckpointTest) PreemptionCheckpointTest.test_error_propagation ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 389, in test_error_propagation mpr.join(timeout=250) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 637, in join self._reraise_if_subprocess_error(process_statuses) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 565, in _reraise_if_subprocess_error six.reraise(*process_status.exc_info) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise raise value File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained return_value = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error self.worker_fn( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 146, in worker_fn strategy = collective_all_reduce_strategy.CollectiveAllReduceStrategy() ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 186, in __init__ CollectiveAllReduceExtended( ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 339, in __init__ self._initialize_strategy(self._cluster_resolver, devices=devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 358, in _initialize_strategy self._initialize_multi_worker(cluster_resolver) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 530, in _initialize_multi_worker context.context().ensure_initialized() ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) ^^^^^^^^^^^^^^^^^ tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server ---------------------------------------------------------------------- Ran 4 tests in 328.072s FAILED (errors=1) ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/ir/tests:testlib.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/ir/tests/testlib.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/ir/tests:testlib.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/ir/tests/testlib.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/ir/tests:testlib.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/ir/tests/testlib.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:no_opt_ops.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/no_opt_ops.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:no_opt_ops.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/no_opt_ops.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:no_opt_ops.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/no_opt_ops.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_asserts.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/convert_asserts.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_asserts.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/convert_asserts.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_asserts.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/convert_asserts.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:module_attributes.hlo.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/module_attributes.hlo.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:module_attributes.hlo.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/module_attributes.hlo.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:module_attributes.hlo.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/module_attributes.hlo.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_and_check_layouts.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_and_check_layouts.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_and_check_layouts.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_and_check_layouts.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_and_check_layouts.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_and_check_layouts.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_send_recv.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_send_recv.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_send_recv.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_send_recv.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_send_recv.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_send_recv.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:send_recv.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/send_recv.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:send_recv.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/send_recv.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:send_recv.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/send_recv.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:module_attributes.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/module_attributes.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:module_attributes.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/module_attributes.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:module_attributes.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/module_attributes.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_disassemble.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_disassemble.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_disassemble.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_disassemble.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_disassemble.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_disassemble.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:ifrt_duplicated_callee_elimination.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/ifrt_duplicated_callee_elimination.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:ifrt_duplicated_callee_elimination.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/ifrt_duplicated_callee_elimination.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:ifrt_duplicated_callee_elimination.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/ifrt_duplicated_callee_elimination.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_while.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_while.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_while.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_while.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_while.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_while.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:input_output_aliasing.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/input_output_aliasing.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:input_output_aliasing.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/input_output_aliasing.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:input_output_aliasing.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/input_output_aliasing.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:add.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/add.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:add.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/add.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:add.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/add.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:non_identity_layouts.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/non_identity_layouts.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:non_identity_layouts.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/non_identity_layouts.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:non_identity_layouts.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/non_identity_layouts.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:ops.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/ops.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:ops.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/ops.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:ops.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/ops.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:snapshot.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/snapshot.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:snapshot.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/snapshot.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:snapshot.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/snapshot.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_launch.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_launch.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_launch.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_launch.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_launch.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_launch.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/ir/tests/ops.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/ir/tests/ops.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/ir/tests/ops.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_case.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_case.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_case.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_case.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_case.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_case.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/framework/tests:outline-with-xla-framework.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/framework/tests/outline-with-xla-framework.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/framework/tests:outline-with-xla-framework.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/framework/tests/outline-with-xla-framework.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/framework/tests:outline-with-xla-framework.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/framework/tests/outline-with-xla-framework.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_concurrent_regions.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/add_concurrent_regions.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_concurrent_regions.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/add_concurrent_regions.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_concurrent_regions.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/add_concurrent_regions.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_outfeed.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_outfeed.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_outfeed.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_outfeed.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_outfeed.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_outfeed.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:legalize_i1_vector_transfers.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/legalize_i1_vector_transfers.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:legalize_i1_vector_transfers.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/legalize_i1_vector_transfers.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:legalize_i1_vector_transfers.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/legalize_i1_vector_transfers.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_op_metadata.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/location_to_op_metadata.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_op_metadata.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/location_to_op_metadata.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_op_metadata.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/location_to_op_metadata.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fusion.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/fusion.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fusion.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/fusion.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fusion.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/fusion.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:case_conditional.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/case_conditional.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:case_conditional.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/case_conditional.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:case_conditional.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/case_conditional.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_gemm.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_gemm.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_gemm.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_gemm.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_gemm.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_gemm.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:spmd_interface_verification.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/spmd_interface_verification.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:spmd_interface_verification.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/spmd_interface_verification.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:spmd_interface_verification.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/spmd_interface_verification.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:simple.hlo.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/simple.hlo.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:simple.hlo.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/simple.hlo.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:simple.hlo.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/simple.hlo.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:dynamic_param.hlo.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/dynamic_param.hlo.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:dynamic_param.hlo.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/dynamic_param.hlo.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:dynamic_param.hlo.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/dynamic_param.hlo.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:entry_computation_layout.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/entry_computation_layout.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:entry_computation_layout.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/entry_computation_layout.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:entry_computation_layout.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/entry_computation_layout.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export-with-layouts.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export-with-layouts.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export-with-layouts.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export-with-layouts.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export-with-layouts.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export-with-layouts.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fully_connected_reference_model.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/fully_connected_reference_model.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fully_connected_reference_model.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/fully_connected_reference_model.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fully_connected_reference_model.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/fully_connected_reference_model.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memset.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_memset.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memset.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_memset.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memset.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_memset.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:stream_assignment.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/stream_assignment.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:stream_assignment.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/stream_assignment.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:stream_assignment.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/stream_assignment.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_hlo_trace.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/add_hlo_trace.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_hlo_trace.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/add_hlo_trace.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_hlo_trace.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/add_hlo_trace.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:passthrough.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/passthrough.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:passthrough.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/passthrough.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:passthrough.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/passthrough.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_attrs.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_attrs.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_attrs.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_attrs.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_attrs.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_attrs.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_replicas.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_replicas.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_replicas.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_replicas.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_replicas.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_replicas.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/import.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/import.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/import.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_custom_call.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_custom_call.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_custom_call.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_custom_call.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_custom_call.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_custom_call.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:rt_to_llvm.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/rt_to_llvm.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:rt_to_llvm.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/rt_to_llvm.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:rt_to_llvm.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/rt_to_llvm.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:rng_bit_generator.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/rng_bit_generator.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:rng_bit_generator.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/rng_bit_generator.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:rng_bit_generator.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/rng_bit_generator.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-operand-with-constant.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-operand-with-constant.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-operand-with-constant.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-operand-with-constant.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-operand-with-constant.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-operand-with-constant.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:bisect.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/bisect.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:bisect.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/bisect.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:bisect.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/bisect.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_fft.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_fft.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_fft.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_fft.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_fft.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_fft.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/memref/transforms/tests:aligned_allocations.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/memref/transforms/tests/aligned_allocations.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/memref/transforms/tests:aligned_allocations.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/memref/transforms/tests/aligned_allocations.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/memref/transforms/tests:aligned_allocations.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/memref/transforms/tests/aligned_allocations.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:bool_compare.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/bool_compare.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:bool_compare.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/bool_compare.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:bool_compare.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/bool_compare.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:ordinal_assignment.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/ordinal_assignment.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:ordinal_assignment.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/ordinal_assignment.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:ordinal_assignment.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/ordinal_assignment.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_loaded_executable.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_loaded_executable.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_loaded_executable.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_loaded_executable.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_loaded_executable.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_loaded_executable.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_custom_calls.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/convert_custom_calls.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_custom_calls.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/convert_custom_calls.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_custom_calls.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/runtime/transforms/tests/convert_custom_calls.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:types.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/types.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:types.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/types.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:types.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/types.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/framework/tests:xla-framework.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/framework/tests/xla-framework.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/framework/tests:xla-framework.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/framework/tests/xla-framework.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/framework/tests:xla-framework.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/framework/tests/xla-framework.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:dynamic.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/dynamic.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:dynamic.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/dynamic.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:dynamic.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/dynamic.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:layouts_and_names.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/layouts_and_names.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:layouts_and_names.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/layouts_and_names.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:layouts_and_names.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/layouts_and_names.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_memref_element_cast_to_llvm.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_memref_element_cast_to_llvm.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_memref_element_cast_to_llvm.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_memref_element_cast_to_llvm.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_memref_element_cast_to_llvm.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_memref_element_cast_to_llvm.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/framework/tests:legalize-xla-framework.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/framework/tests/legalize-xla-framework.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/framework/tests:legalize-xla-framework.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/framework/tests/legalize-xla-framework.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/framework/tests:legalize-xla-framework.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/framework/tests/legalize-xla-framework.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:hlo_text_to_lhlo_no_opt.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/hlo_text_to_lhlo_no_opt.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:hlo_text_to_lhlo_no_opt.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/hlo_text_to_lhlo_no_opt.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:hlo_text_to_lhlo_no_opt.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/hlo_text_to_lhlo_no_opt.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:truncate-function.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/truncate-function.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:truncate-function.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/truncate-function.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:truncate-function.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/truncate-function.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/xla_cpu/tests:invalid.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/xla_cpu/tests/invalid.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/xla_cpu/tests:invalid.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/xla_cpu/tests/invalid.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/xla_cpu/tests:invalid.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/xla_cpu/tests/invalid.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:remove_copies_to_out_params.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/remove_copies_to_out_params.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:remove_copies_to_out_params.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/remove_copies_to_out_params.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:remove_copies_to_out_params.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/remove_copies_to_out_params.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:reduce-scf-forall-bounds.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/reduce-scf-forall-bounds.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:reduce-scf-forall-bounds.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/reduce-scf-forall-bounds.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:reduce-scf-forall-bounds.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/reduce-scf-forall-bounds.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:location.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/location.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:location.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/location.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:location.hlotxt.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/location.hlotxt.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:library_ops_to_cpu_runtime.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/library_ops_to_cpu_runtime.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:library_ops_to_cpu_runtime.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/library_ops_to_cpu_runtime.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:library_ops_to_cpu_runtime.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/library_ops_to_cpu_runtime.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_call.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_call.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_call.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call_loaded_executable.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_call_loaded_executable.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call_loaded_executable.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_call_loaded_executable.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ ==================== Test output for //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call_loaded_executable.mlir.test: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_call_loaded_executable.mlir.test.runfiles/llvm-project/llvm/utils/lit/lit.py", line 3, in from lit.main import main ModuleNotFoundError: No module named 'lit' ================================================================================ //tensorflow/c:c_api_experimental_test PASSED in 34.9s //tensorflow/c:c_api_function_test PASSED in 32.7s //tensorflow/c:c_api_test_cpu PASSED in 38.0s //tensorflow/c:c_test PASSED in 34.4s //tensorflow/c:env_test_cpu PASSED in 31.3s //tensorflow/c:kernels_test_cpu PASSED in 45.4s //tensorflow/c:ops_test PASSED in 27.2s //tensorflow/c:tf_status_helper_test PASSED in 0.1s //tensorflow/c:while_loop_test PASSED in 40.9s //tensorflow/c/eager:c_api_cluster_test_cpu PASSED in 30.1s //tensorflow/c/eager:c_api_remote_function_test_cpu PASSED in 36.2s //tensorflow/c/eager:c_api_remote_test_cpu PASSED in 34.2s //tensorflow/c/eager:c_api_test_cpu PASSED in 36.8s //tensorflow/c/eager:custom_device_test PASSED in 34.6s //tensorflow/c/eager/parallel_device:parallel_device_lib_test PASSED in 32.2s //tensorflow/c/eager/parallel_device:parallel_device_remote_test PASSED in 38.6s //tensorflow/c/eager/parallel_device:parallel_device_test PASSED in 29.5s //tensorflow/c/experimental/filesystem/plugins/gcs:expiring_lru_cache_test PASSED in 0.1s //tensorflow/c/experimental/filesystem/plugins/gcs:ram_file_block_cache_test PASSED in 2.5s //tensorflow/c/experimental/grappler:grappler_test PASSED in 26.6s //tensorflow/c/experimental/next_pluggable_device:tensor_pjrt_buffer_util_test PASSED in 7.2s //tensorflow/c/experimental/ops/gen/common:case_format_test PASSED in 0.6s //tensorflow/c/experimental/ops/gen/cpp:cpp_generator_test PASSED in 1.0s //tensorflow/c/experimental/ops/gen/cpp/renderers:renderer_test PASSED in 0.6s //tensorflow/c/experimental/saved_model/core:constant_loading_test PASSED in 20.4s //tensorflow/c/experimental/saved_model/core:object_graph_traversal_test PASSED in 10.6s //tensorflow/c/experimental/saved_model/core:saved_variable_loading_test PASSED in 22.7s //tensorflow/c/experimental/saved_model/core:signature_flattening_test PASSED in 11.9s //tensorflow/c/experimental/saved_model/core:tf_concrete_function_loading_test PASSED in 12.7s //tensorflow/c/experimental/saved_model/core/ops:restore_ops_test PASSED in 17.7s //tensorflow/c/experimental/saved_model/core/ops:variable_ops_test PASSED in 18.5s //tensorflow/c/experimental/saved_model/internal:saved_model_api_test PASSED in 37.1s //tensorflow/c/experimental/stream_executor:stream_executor_test PASSED in 0.3s //tensorflow/c/kernels:bitcast_op_test PASSED in 0.6s //tensorflow/c/kernels:summary_op_benchmark_test PASSED in 0.7s //tensorflow/c/kernels:summary_op_test PASSED in 0.8s //tensorflow/c/kernels:tensor_shape_utils_test PASSED in 0.1s //tensorflow/cc:cc_op_gen_test PASSED in 0.4s //tensorflow/cc:client_client_session_test PASSED in 2.4s //tensorflow/cc:coordinator_test PASSED in 3.7s //tensorflow/cc:framework_cc_ops_test PASSED in 2.0s //tensorflow/cc:framework_gradient_checker_test PASSED in 2.7s //tensorflow/cc:framework_gradients_test PASSED in 4.1s //tensorflow/cc:framework_scope_test PASSED in 0.8s //tensorflow/cc:framework_while_gradients_test PASSED in 2.6s //tensorflow/cc:gradients_array_grad_test PASSED in 6.9s //tensorflow/cc:gradients_data_flow_grad_test PASSED in 2.9s //tensorflow/cc:gradients_functional_grad_test PASSED in 2.3s //tensorflow/cc:gradients_image_grad_test PASSED in 5.0s //tensorflow/cc:gradients_linalg_grad_test PASSED in 2.6s //tensorflow/cc:gradients_manip_grad_test PASSED in 1.9s //tensorflow/cc:gradients_math_grad_test PASSED in 9.8s //tensorflow/cc:gradients_nn_grad_test PASSED in 3.4s //tensorflow/cc:gradients_resource_variable_grad_test PASSED in 2.2s //tensorflow/cc:ops_const_op_test PASSED in 1.0s //tensorflow/cc:ops_while_loop_test PASSED in 2.0s //tensorflow/cc:queue_runner_test PASSED in 12.6s //tensorflow/cc/experimental/base/tests:tensor_test PASSED in 0.1s //tensorflow/cc/experimental/base/tests:tensorhandle_test PASSED in 32.1s //tensorflow/cc/experimental/libexport:load_test PASSED in 0.1s //tensorflow/cc/experimental/libexport:save_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_module_test PASSED in 30.3s //tensorflow/cc/experimental/libtf:libtf_object_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_perf_test PASSED in 0.2s //tensorflow/cc/experimental/libtf:libtf_runtime_test PASSED in 32.9s //tensorflow/cc/experimental/libtf:libtf_transform_test PASSED in 28.8s //tensorflow/cc/experimental/libtf:libtf_value_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_visit_test PASSED in 0.2s //tensorflow/cc/experimental/libtf/impl:iostream_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:none_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:scalars_test PASSED in 0.4s //tensorflow/cc/experimental/libtf/impl:string_test PASSED in 0.2s //tensorflow/cc/experimental/libtf/impl:tensor_spec_test PASSED in 0.2s //tensorflow/cc/saved_model:bundle_v2_test PASSED in 0.2s //tensorflow/cc/saved_model:fingerprinting_test PASSED in 1.5s //tensorflow/cc/saved_model:metrics_test PASSED in 0.2s //tensorflow/cc/saved_model:reader_test PASSED in 0.2s //tensorflow/cc/saved_model:saved_model_bundle_lite_test PASSED in 6.7s //tensorflow/cc/saved_model:saved_model_bundle_test PASSED in 9.3s //tensorflow/cc/saved_model:util_test PASSED in 0.1s //tensorflow/cc/saved_model/experimental/tests:saved_model_api_test PASSED in 33.1s //tensorflow/cc/tools:freeze_saved_model_test PASSED in 2.0s //tensorflow/compiler/aot:codegen_test PASSED in 26.0s //tensorflow/compiler/jit:compilability_check_util_test PASSED in 20.5s //tensorflow/compiler/jit:deadness_analysis_test PASSED in 12.7s //tensorflow/compiler/jit:device_compilation_cache_test PASSED in 6.4s //tensorflow/compiler/jit:device_compilation_cluster_signature_test PASSED in 4.7s //tensorflow/compiler/jit:device_compilation_profiler_test PASSED in 22.7s //tensorflow/compiler/jit:device_compiler_client_test PASSED in 5.2s //tensorflow/compiler/jit:device_compiler_disable_test PASSED in 27.8s //tensorflow/compiler/jit:device_executable_persistor_test PASSED in 25.0s //tensorflow/compiler/jit:device_util_test PASSED in 5.6s //tensorflow/compiler/jit:encapsulate_util_test PASSED in 0.7s //tensorflow/compiler/jit:node_matchers_test PASSED in 1.1s //tensorflow/compiler/jit:resource_operation_safety_analysis_test PASSED in 10.0s //tensorflow/compiler/jit:shape_inference_test PASSED in 0.8s //tensorflow/compiler/jit:xla_activity_listener_test PASSED in 23.5s //tensorflow/compiler/jit:xla_cluster_util_test PASSED in 9.3s //tensorflow/compiler/jit:xla_compile_util_test PASSED in 5.2s //tensorflow/compiler/jit:xla_kernel_creator_test PASSED in 10.0s //tensorflow/compiler/jit:xla_launch_util_test PASSED in 22.0s //tensorflow/compiler/jit/tests:auto_clustering_test PASSED in 25.0s //tensorflow/compiler/mlir:mlir_graph_optimization_pass_test PASSED in 14.9s //tensorflow/compiler/mlir:register_common_dialects_test PASSED in 17.1s //tensorflow/compiler/mlir/lite:lstm_utils_test PASSED in 0.7s //tensorflow/compiler/mlir/lite:perception_ops_utils_test PASSED in 0.6s //tensorflow/compiler/mlir/lite:size_utils_test PASSED in 0.5s //tensorflow/compiler/mlir/lite:tftext_utils_test PASSED in 0.5s //tensorflow/compiler/mlir/lite/experimental/remat:rematerializer_test PASSED in 1.0s //tensorflow/compiler/mlir/lite/experimental/tac:execution_metadata_exporter_test PASSED in 4.1s //tensorflow/compiler/mlir/lite/experimental/tac/tests:compute-cost.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-gpu.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-nnapi.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/experimental/tac/tests:fold-constants-to-subgraph.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-alternative-subgraph.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-op-cost.mlir.test PASSED in 2.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests:pick-subgraphs.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/experimental/tac/tests:raise-target-subgraphs.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/experimental/tac/tests:tac-filter.mlir.test PASSED in 2.9s //tensorflow/compiler/mlir/lite/experimental/tac/tests:target-annotation.mlir.test PASSED in 2.6s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:device-transform-nnapi.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:simple-graph.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/metrics:error_collector_inst_test PASSED in 0.6s //tensorflow/compiler/mlir/lite/quantization:numerical_utils_test PASSED in 0.2s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_model_test PASSED in 13.1s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_weights_test PASSED in 11.6s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_default.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_legacy.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant_4bit.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/quantization/tests:import_quant_stats.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/sparsity:sparsify_model_test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:compose-uniform-quantized-type.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:fold_broadcast.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:fuse_mhlo_convolution.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-inplaceupdate.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-skip-quantization-ops.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tf-fb-tf.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-add.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-broadcast_in_dim.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-clamp.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-compare.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-concat.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-constant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-conv.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-dot.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-gather.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-max.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-mul.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-pad.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-reshape.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-rsqrt.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-scatter.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-sub.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-add.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-broadcast.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-clamp.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-concat.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-constant.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-conv.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-max.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-mul.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-pad.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-reshape.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-rsqrt.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-sub.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-allow-tf.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-smuggle-resize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:optimize.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-clamp.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-concat.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-conv.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-division.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-logistic.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-multiply.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-reduce-window.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-resize-bilinear.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-subtract.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-tf-quantize.mlir.test PASSED in 4.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:unfuse_mhlo_batch_norm.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:uniform-quantized-stablehlo-to-tfl.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:analyze-variables.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:canonicalize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:decompose-hybrid-quantization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:default_quant_params.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:dilated-conv.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:fuse-tftext.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:get-arithmetic-count.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:guarantee_func_has_one_use.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:inlining.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:insert_call_once_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:legalize-tensorlist.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:legalize-tf-assert.mlir.test PASSED in 2.7s //tensorflow/compiler/mlir/lite/tests:legalize-tf-hashtables.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:legalize-tf-no-runtime-verification.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:legalize-tf-variables.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:legalize-tf-while.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:legalize-tf.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/tests:legalize_jax_random.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:lift_tflite_flex_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-default-to-single-batch.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-enable-dynamic-update-slice.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests:modify_io_nodes.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:optimize-after-quantization.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:optimize.mlir.test PASSED in 3.1s //tensorflow/compiler/mlir/lite/tests:optimize_functional_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:optimize_no_verify.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:optimize_op_order.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:partitioned-topological-sort.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:pin-ops-with-side-effects.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:post-quantize-dynamic-range.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:post-quantize.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:prepare-composite-functions-tf.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-dynamic-range.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training-16bits.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-signed.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:prepare-quantize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant-4bit.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:prepare-tf-with-allowing-bf16-and-f16-type-legalization.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:prepare-tf.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:quantize-dynamic-range.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests:quantize-numeric-verify.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:quantize-variables.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:quantize.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests:raise-custom-ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:reduce_while_operands.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:shape-inference.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:split-merged-operands.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:tfl_while_op_licm.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:tfl_while_outline.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests:trim-functions-tf.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:unfold-large-splat-constant.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.line.part.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.stack.part.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/end2end:add.pbtxt.test PASSED in 2.3s //tensorflow/compiler/mlir/lite/tests/end2end:back2back_fake_quant.pbtxt.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/tests/end2end:control_flow_v1.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d_nchw.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:custom_opdef.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/end2end:disallow_stateful_partitioned_call.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel_4bit.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity.pbtxt.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity_4bit.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/end2end:graph-input-node.pbtxt.test PASSED in 14.0s //tensorflow/compiler/mlir/lite/tests/end2end:graph_with_placeholder_with_default.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:if_op.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:quant_stats.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul_disabled.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:basic_lstm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:bucketize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants_offset.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:control_edges.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op_offset.mlir.test PASSED in 8.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:dynamic_shape.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:empty_input_output_names.json.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:external_constant.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:if_op.mlir.test PASSED in 2.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:import_json.json.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_arrays.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_output_names_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:legacy_reshape.json.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.json.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:many_attribute_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:math.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:matmul.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:multi_output_op.json.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional_input.json.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:output_arrays.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning_function_input_as_output.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quant_stats.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quantization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:reshape.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature_with_multiple_entry_points.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:simple.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:tf_variant_type.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_function_output.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_tensor.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:while_op.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2exec:tfl_while_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:basic_lstm.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:bucketize.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_op_with_tflite_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_tensorlist_reserve.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d_v2.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_builtin.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_custom.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex_enable_builtin.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:dynamic_shape_constant.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fake_quant.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_exclusively.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_complex128.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_f64.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_tflite_op.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected_v2.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:hashtable_resource.mlir.test PASSED in 4.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:if_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:logical.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:low_bit_packing.mlir.test PASSED in 16.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_asym_attr.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_quantized.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:math.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:metadata.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v2.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v3.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:nn.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:numeric_verify.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:optional.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:quantization.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:reshape.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_output_override.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_multiple_entry_points.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_no_inputs.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_connected_control_nodes.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_unconnected_control_nodes.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf_v2.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tf_entry_function.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tfl_while_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:transpose_conv_optional.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:type_attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_lstm.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_rnn.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unranked_tensor.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unsorted_segment_prod.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_func.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:while_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/stablehlo:convert_tf_quant_to_mhlo_int_test PASSED in 19.0s //tensorflow/compiler/mlir/quantization/stablehlo:math_utils_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/stablehlo/tests:fill_quantization_options_test PASSED in 2.2s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibrator_singleton_test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:custom_aggregator_op_test PASSED in 22.4s //tensorflow/compiler/mlir/quantization/tensorflow/cc:const_op_size_test PASSED in 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/cc:constant_fold_test PASSED in 33.6s //tensorflow/compiler/mlir/quantization/tensorflow/cc:convert_asset_args_test PASSED in 6.5s //tensorflow/compiler/mlir/quantization/tensorflow/cc:save_variables_test PASSED in 0.5s //tensorflow/compiler/mlir/quantization/tensorflow/cc:status_macro_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/tensorflow/debugging:mlir_dump_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/tensorflow/python:concurrency_test PASSED in 56.2s //tensorflow/compiler/mlir/quantization/tensorflow/python:pywrap_quantize_model_test PASSED in 24.7s //tensorflow/compiler/mlir/quantization/tensorflow/python:representative_dataset_test PASSED in 13.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:cast_bf16_ops_to_f32.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_custom_aggregation_op_to_quant_stats.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_fake_quant_to_qdq.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tpu_model_to_cpu.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:duplicate_shape_determining_constants.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_flow.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_xla.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_custom_aggregation_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_main_function.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_drq.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_weight_only.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_restore_op.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_save_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:issue_ids_of_custom_aggregation_ops.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_hashtable_ops_as_args.mlir.test PASSED in 2.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq_min_elements.mlir.test PASSED in 4.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_xla.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:mark_functions_noinline.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_duplicate_resource_ops.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_initializer_function_ops_to_main.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_save_function_ops_to_main.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:optimize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_lifting.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq_per_channel.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq_per_channel.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op_weight_only.mlir.test PASSED in 2.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_drq.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_weight_only.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_xla.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_drq.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_xla.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:remove_var_init_by_const.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops_large_constants.mlir.test PASSED in 15.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:unfreeze_constants.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/utils:tf_to_xla_attribute_utils_test PASSED in 39.6s //tensorflow/compiler/mlir/stablehlo:stablehlo_test PASSED in 0.1s //tensorflow/compiler/mlir/tensorflow:bridge_logger_test PASSED in 6.3s //tensorflow/compiler/mlir/tensorflow:call_graph_util_test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow:cluster_util_test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow:convert_tensor_test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow:convert_type_test PASSED in 0.2s //tensorflow/compiler/mlir/tensorflow:device_util_test PASSED in 0.3s //tensorflow/compiler/mlir/tensorflow:dump_graph_test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow:dump_mlir_util_test PASSED in 14.9s //tensorflow/compiler/mlir/tensorflow:error_util_test PASSED in 0.1s //tensorflow/compiler/mlir/tensorflow:tf_mlir_translate_registration_test PASSED in 20.8s //tensorflow/compiler/mlir/tensorflow:tf_saved_model_test PASSED in 0.4s //tensorflow/compiler/mlir/tensorflow:tpu_rewrite_device_util_test PASSED in 0.4s //tensorflow/compiler/mlir/tensorflow:xla_rewrite_util_test PASSED in 0.3s //tensorflow/compiler/mlir/tensorflow/tests:add_functions_for_exported_names.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:annotate-parameter-replication.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:batchmatmul_to_einsum.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:breakup-islands.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:cannonicalize_ops_outside_compilation.mlir.test PASSED in 2.8s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize_compile_and_replicate_attributes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:check_control_dependencies.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:cluster_formation.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:cluster_ops_by_policy.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:cluster_outlining.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests:cluster_tf_ops_pass.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:constant-fold.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:constant_op_device_assignment.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:convert-tf-control-flow-to-scf.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:convert_control_to_data_outputs.mlir.test PASSED in 2.8s //tensorflow/compiler/mlir/tensorflow/tests:convert_launch_func_to_tf_call.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:convert_session_initializer_to_function.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:convert_to_legacy_compile_and_replicate_attributes.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:decompose_reduce_dataset.mlir.test PASSED in 6.9s //tensorflow/compiler/mlir/tensorflow/tests:decompose_resource_ops.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment_by_func_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:device_attribute_to_launch.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:device_canonicalize.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:device_copy.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:drop_while_shape_invariant.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:einsum.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:embedding_pipelining.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:embedding_program_key.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:embedding_sequencing.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:empty-main.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:end-to-end-tpu-reshard-variables.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:executor_canonicalize.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_coarsening.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_materialize_const.mlir.test PASSED in 17.9s //tensorflow/compiler/mlir/tensorflow/tests:extract_head_tail_outside_compilation.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:extract_outside_compilation.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:extract_tpu_copy_with_dynamic_shape_op.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:fold-broadcast.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:freeze_variables.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:func-attr-invalid.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:func-attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-cfg.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-regions.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if-fail.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:fused_kernel_matcher.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:gpu_fusion.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning_preserve_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:group_by_dialect.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:guarantee-all-funcs-one-use.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:hoist_loop_invariant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:hoist_replicate_invariant_resource_writes.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:host_launch_to_outside_compiled.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_invalid.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_saved_model.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:inlining.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:isolate-placer.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:launch_outlining.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute_legacy.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_60.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_70.mlir.test PASSED in 2.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nchw.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nhwc.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_begin.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_end.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nchw.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nhwc.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:legalize_hlo.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_arg_control_dep.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_with_control_flow.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:localize_var_handles.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program_invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:lower_quantized.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:lower_tf.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:lower_variable_ops_to_ml_program.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:mark_input_output_aliases.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:mark_ops_for_outside_compilation.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:materialize_passthrough_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:merge_control_flow.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:mlprogram.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:name_anonymous_iterators.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:optimize-arg-operand-constraint.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:optimize.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:order_by_dialect.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:outside_compiled_to_host_launch.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands_legacy.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:prepare_tpu_computation_for_tf_export.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args_functions.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:promote_var_handles_to_args.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:readonly_references_to_resources.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:region-control-flow-to-functional.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_arguments.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_while_results.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:replica_id_to_device_ordinal.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:replicate_invariant_op_hoisting.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:replicate_tensor_list_init_ops.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island_legacy.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:resource-alias-analysis-test.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:resource-device-inference.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:resource_analyzer.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:resource_inlining.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:resource_op_lifting.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:rewrite_tpu_embedding_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:roundtrip-tf-executor.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:shape_inference.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:side-effect-analysis-test.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:sink_constant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:split_into_island_per_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:stack_ops_decomposition.mlir.test PASSED in 8.9s //tensorflow/compiler/mlir/tensorflow/tests:strip_noinline.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:strip_saved_module_metadata.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:strip_tf_attributes.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tensor_array_ops_decomposition.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tensor_list_ops_decomposition.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:tf-executor-to-functional.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf-functional-to-executor.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf-ops.mlir.test PASSED in 2.8s //tensorflow/compiler/mlir/tensorflow/tests:tf-reduce-identity.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_map_and_batch.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_pmap_and_batch.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_index_selector.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops_invalid.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_invalid.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_location_roundtrip.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_printer.mlir.test PASSED in 14.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_side_effect.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_optimize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_asset_sinking.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_deduplicate_bound_input_bindings.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_assets.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors_mutable_tensors.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init_fail.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables.mlir.test PASSED in 3.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables_invalid_session.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_mark_initialized_variables.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops_invalid.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors_interprocedural.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_remove_vars_in_session_initializer.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_side_effect.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_trait_folds.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tfrt_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu-annotate-dynamic-shape-inputs.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu-cluster-cleanup-attributes.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu-dynamic-layout-pass.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu-merge-variables-with-execute.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu-multiple-while-body-func.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu-resource-read-for-write.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu-variable-runtime-reformatting.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_cluster_formation.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_composite_resource_ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_splits.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_device_propagation.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_host_computation_expansion.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_identity_pruning.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu_parallel_execute_sink_resource_write.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_partitioned_op_conversion.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_reorder_replicate_and_partitioned_inputs.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tpu_resource_partitioning.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_rewrite.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_sharding_identification.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_space_to_depth_pass.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_tail_with_tobool_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_update_embedding_enqueue_op_inputs.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_validate_inputs.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:transpose-op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:unroll-batch-matmul.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:update_control_dependencies.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:warn_when_using_deprecated_dumps.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:while_licm.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_deserialization.mlir.test PASSED in 14.7s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_round_trip.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_serialization.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:xla_cluster_formation.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:xla_inline_device_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite_v2.mlir.test PASSED in 3.1s //tensorflow/compiler/mlir/tensorflow/tests:xla_validate_iputs.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:add.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding-invalid.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding-hook.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:convert_mhlo_quant_to_int.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.mlir.test PASSED in 3.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:mlir-module-serialized-str-attr.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:replicate-tensor-list-init-ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:result-sharding.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr-invalid.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr.mlir.test PASSED in 14.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference-after-legalization.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:stablehlo_add.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:executor_tpuv1_island_coarsening.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:while_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:executor_tpuv1_inline_tpu_island.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:while_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:case_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:executor_tpuv1_outline_tpu_island.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:while_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:add.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-as-fetch.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-control-dep.pbtxt.test PASSED in 2.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type-with-subtype.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-multi-data-type-with-subtype.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-retval-attrs.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:case_op.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:const-values.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:device-arg-retval-attr.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-input-shapes.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-value-attr.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-as-fetch.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-control-dep.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:force_shared_name_for_resource_ops.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:function-func-attr.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-if-ops.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-while-ops.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-control-ret.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-retval-of-arg.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-custom-operation.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-default-attr.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-device-retval.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-empty-tensor-content.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-func-attr.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-call.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-diff-island.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-same-island.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-defs.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-input-shapes.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-name-bug.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-resource-args.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-gradient-def.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-input-func-arg-name-collision.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-library.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-malformed.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-scalar-input.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-uint8-return.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-undefined-output.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-version-info.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-while-loop.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:invalid-output-index.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:legacy-fed-input-without-inputs.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:merge_node_with_function.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:mlir_passthrough_op.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multi-output-feeds.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multiple-use-next-iteration.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:node-locations.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes-attr.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example.pbtxt.test PASSED in 2.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example_v2.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:partial-device-name.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:prune_unused_nodes.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:quint8-const.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:shape-attrs.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:stateful-attribute.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:string-attr.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:switch_n.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:target.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tensor-list.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tf-data-pipeline.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:unregistered_kernel.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/batch_use_same_function:saved_model.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graph:convert_tensor.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:aliasing_arg_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:case.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:convert_tensor.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_shape_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_size_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:device-arg-retval-attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:export_main_to_flib.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:fetch_feed_names.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_list_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-control-ret.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-order.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args-handle-info.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args.mlir.test PASSED in 3.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-if-ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-while-ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:graph-as-function.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:infer_derived_attribute.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:invalid_input.mlir.test PASSED in 3.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:legalized_name.mlir.test PASSED in 2.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:missing-main.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:noop.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:optional_symbol_ref.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:output-shapes-attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example_v2.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:preserve-entry-func-names.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-type-attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-while-loop.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:shape_list_attr.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple_tf_dialect_op.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:stringescape.mlir.test PASSED in 2.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:switchn.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-gradient-attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-legacy-call.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_add.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_identity_n.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_tpu_embedding_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_list_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_name.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_output_name.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:while-loop.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/tf_to_hlo_pipeline:sccp-post-shape-inference.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/tpu_bridge_v1:end_to_end.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/api/v0:compile_mlir_util_test PASSED in 5.6s //tensorflow/compiler/mlir/tf2xla/api/v0:compile_tf_graph_test PASSED in 0.2s //tensorflow/compiler/mlir/tf2xla/api/v1:legalize_tf_test PASSED in 23.3s //tensorflow/compiler/mlir/tf2xla/internal:mlir_pass_instrumentation_test PASSED in 6.3s //tensorflow/compiler/mlir/tf2xla/tests:adjust-layout.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_runtime_pipeline.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_sparsification.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-BatchMatMulV2.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-binary-elementwise.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-collective.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-communication.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-include-tf2xla-fallback.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-no-tf2xla-fallback.mlir.test PASSED in 4.1s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-prefer-tf2xla.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-types.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-with-tf2xla-hlo-importer.mlir.test PASSED in 3.0s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf.mlir.test PASSED in 10.9s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_cpu.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_gpu.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization-no-chlo.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/transforms:legalization_op_config_test PASSED in 30.6s //tensorflow/compiler/mlir/tf2xla/transforms:legalize_tf_types_test PASSED in 15.6s //tensorflow/compiler/mlir/tf2xla/transforms:tf2xla_rewriter_test PASSED in 20.2s //tensorflow/compiler/mlir/tf2xla/transforms:verify_tfxla_legalization_test PASSED in 17.4s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_targets_test PASSED in 0.6s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_tf_test PASSED in 3.1s //tensorflow/compiler/mlir/tfr:graph_decompose_test PASSED in 26.0s //tensorflow/compiler/mlir/tfr:node_expansion_test PASSED in 9.5s //tensorflow/compiler/mlir/tfr:op_reg_gen_test PASSED in 34.4s //tensorflow/compiler/mlir/tfr:tfr_decompose_ctx_test PASSED in 6.1s //tensorflow/compiler/mlir/tfr:tfr_gen_test PASSED in 43.4s //tensorflow/compiler/mlir/tfr/examples/customization:test_ops_test PASSED in 26.1s //tensorflow/compiler/mlir/tfr/examples/mnist:mnist_ops_test PASSED in 28.0s //tensorflow/compiler/mlir/tfr/examples/pad:pad_ops_test PASSED in 26.6s //tensorflow/compiler/mlir/tfrt/tests:batch_function_fallback_resource_variable_as_captured_tensor.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests:batch_function_lowering.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests:convert_ref_variables.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests:cross_device_transfer.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests:deduplicate_if_results.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests:fuse_tpu_compile_and_execute_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests:hoist_invariant_ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests:hoist_invariant_ops_mlrt.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests:optimize.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tfrt/tests:remove_device_attribute.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests:sink_in_invariant_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tfrt/tests:xla_launch_fallback.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests:xla_rewrite.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/analysis:cost_analysis.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/analysis:tensor_array_side_effect_analysis.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/analysis:update_op_cost_in_tfrt_mlir_test PASSED in 0.5s //tensorflow/compiler/mlir/tfrt/tests/ir:fallback_opt.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/ir:tfrt_fallback_util_test PASSED in 0.4s //tensorflow/compiler/mlir/tfrt/tests/mlrt:assign_op_key.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/mlrt:fuse_mlrt_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/mlrt:inline.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tfrt/tests/mlrt:parallelization.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/mlrt:tf_to_mlrt.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tfrt/tests/mlrt:tpu_conversions.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/mlrt:while_to_map_fn.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:attributes.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:basic.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:batch_function_deduplicate.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:batch_function_deduplicate_failed.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:const_tensor.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:control_flow.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:decompose_resource_op.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:derived_attrs.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:device_conversion.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:errors.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback.mlir.test PASSED in 3.0s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback_canonicalization.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback_inline.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_attributes.mlir.test PASSED in 3.1s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_attributes_multiple_callers.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_use_fallback_tensor.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:insert_fallback_tensor_copy.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:merge_tf_if_ops.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:optimize_tf_control_flow_side_effect.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:remove_tf_if_const_args.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:reorder_assert.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:side_effects.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:tf_to_corert_pipeline.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:tf_to_corert_pipeline_refvar.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:whileop.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/translate/mlrt:mlir_to_bytecode_test PASSED in 0.3s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_deallocation.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_reuse.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tools/kernel_gen/tests:bufferize.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tools/kernel_gen/tests:copy_cleanup.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:embed_tf_framework.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:invalid.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:isinf.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:ops.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/tools/kernel_gen/tests:parallel_loops_to_sequential.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:rewrite_tf_framework_assert.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tanh.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf-legalize-to-lmhlo.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_abi_knowledge.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_framework_legalize_to_llvm.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_kernel_gpu_launch_to_llvm.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_to_jit_invocations.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tosa/tests:convert-tfl-uint8.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:convert_metadata.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tosa/tests:fuse-bias-tf.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tosa/tests:lower-complex-types.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:lower_global_tensors.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tosa/tests:multi_add.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tosa/tests:retain_call_once_funcs.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:strip-quant-types.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:strip_metadata.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:tf-tfl-to-tosa-pipeline.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tosa/tests:tf-to-tosa-pipeline.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-dequantize_softmax.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline-filtered.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline.mlir.test PASSED in 5.4s //tensorflow/compiler/mlir/tosa/tests:verify_fully_converted.mlir.test PASSED in 1.1s //tensorflow/compiler/tests:adadelta_test_cpu PASSED in 17.8s //tensorflow/compiler/tests:adagrad_da_test_cpu PASSED in 12.6s //tensorflow/compiler/tests:adagrad_test_cpu PASSED in 11.8s //tensorflow/compiler/tests:adam_test_cpu PASSED in 15.5s //tensorflow/compiler/tests:add_n_test_cpu PASSED in 15.2s //tensorflow/compiler/tests:argminmax_test_cpu PASSED in 23.8s //tensorflow/compiler/tests:argminmax_test_cpu_mlir_bridge_test PASSED in 18.1s //tensorflow/compiler/tests:bucketize_op_test_cpu PASSED in 9.7s //tensorflow/compiler/tests:bucketize_op_test_cpu_mlir_bridge_test PASSED in 9.0s //tensorflow/compiler/tests:case_test_cpu PASSED in 13.7s //tensorflow/compiler/tests:cast_ops_test_cpu PASSED in 9.2s //tensorflow/compiler/tests:cast_ops_test_cpu_mlir_bridge_test PASSED in 10.2s //tensorflow/compiler/tests:categorical_op_test_cpu PASSED in 14.4s //tensorflow/compiler/tests:categorical_op_test_cpu_mlir_bridge_test PASSED in 14.2s //tensorflow/compiler/tests:cholesky_op_test_cpu PASSED in 18.1s //tensorflow/compiler/tests:cholesky_op_test_cpu_mlir_bridge_test PASSED in 26.2s //tensorflow/compiler/tests:clustering_test_cpu PASSED in 11.0s //tensorflow/compiler/tests:clustering_test_cpu_mlir_bridge_test PASSED in 10.3s //tensorflow/compiler/tests:concat_ops_test_cpu PASSED in 17.5s //tensorflow/compiler/tests:concat_ops_test_cpu_mlir_bridge_test PASSED in 11.4s //tensorflow/compiler/tests:cond_test_cpu PASSED in 10.4s //tensorflow/compiler/tests:const_arg_test_cpu PASSED in 10.2s //tensorflow/compiler/tests:const_test_cpu PASSED in 11.3s //tensorflow/compiler/tests:data_format_ops_test_cpu PASSED in 13.4s //tensorflow/compiler/tests:data_format_ops_test_cpu_mlir_bridge_test PASSED in 16.0s //tensorflow/compiler/tests:dense_layer_test_cpu PASSED in 18.4s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu PASSED in 29.3s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu_mlir_bridge_test PASSED in 17.2s //tensorflow/compiler/tests:dynamic_stitch_test_cpu PASSED in 10.1s //tensorflow/compiler/tests:dynamic_stitch_test_cpu_mlir_bridge_test PASSED in 9.9s //tensorflow/compiler/tests:eager_test_cpu PASSED in 28.9s //tensorflow/compiler/tests:einsum_op_test_cpu PASSED in 11.1s //tensorflow/compiler/tests:einsum_op_test_cpu_mlir_bridge_test PASSED in 13.1s //tensorflow/compiler/tests:ensure_shape_op_test_cpu PASSED in 10.8s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu PASSED in 10.4s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu_mlir_bridge_test PASSED in 10.6s //tensorflow/compiler/tests:fake_quant_ops_test_cpu PASSED in 16.1s //tensorflow/compiler/tests:fake_quant_ops_test_cpu_mlir_bridge_test PASSED in 14.4s //tensorflow/compiler/tests:fifo_queue_test_cpu PASSED in 11.9s //tensorflow/compiler/tests:fifo_queue_test_cpu_mlir_bridge_test PASSED in 35.3s //tensorflow/compiler/tests:ftrl_ops_test_cpu PASSED in 13.1s //tensorflow/compiler/tests:ftrl_ops_test_cpu_mlir_bridge_test PASSED in 13.7s //tensorflow/compiler/tests:function_test_cpu PASSED in 10.2s //tensorflow/compiler/tests:function_test_cpu_mlir_bridge_test PASSED in 15.4s //tensorflow/compiler/tests:gather_nd_op_test_cpu PASSED in 11.2s //tensorflow/compiler/tests:gather_nd_op_test_cpu_mlir_bridge_test PASSED in 13.9s //tensorflow/compiler/tests:gather_test_cpu PASSED in 41.3s //tensorflow/compiler/tests:gather_test_cpu_mlir_bridge_test PASSED in 49.2s //tensorflow/compiler/tests:jit_test_cpu PASSED in 58.7s //tensorflow/compiler/tests:listdiff_op_test_cpu PASSED in 12.8s //tensorflow/compiler/tests:listdiff_op_test_cpu_mlir_bridge_test PASSED in 16.1s //tensorflow/compiler/tests:lrn_ops_test_cpu PASSED in 9.8s //tensorflow/compiler/tests:lrn_ops_test_cpu_mlir_bridge_test PASSED in 10.3s //tensorflow/compiler/tests:lstm_test_cpu PASSED in 26.5s //tensorflow/compiler/tests:manip_ops_test_cpu PASSED in 15.0s //tensorflow/compiler/tests:manip_ops_test_cpu_mlir_bridge_test PASSED in 19.2s //tensorflow/compiler/tests:matrix_band_part_test_cpu PASSED in 39.5s //tensorflow/compiler/tests:matrix_band_part_test_cpu_mlir_bridge_test PASSED in 33.7s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu PASSED in 25.5s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu_mlir_bridge_test PASSED in 21.5s //tensorflow/compiler/tests:matrix_solve_op_test_cpu PASSED in 35.9s //tensorflow/compiler/tests:matrix_solve_op_test_cpu_mlir_bridge_test PASSED in 16.4s //tensorflow/compiler/tests:matrix_triangular_solve_op_test_cpu PASSED in 25.7s //tensorflow/compiler/tests:matrix_triangular_solve_op_test_cpu_mlir_bridge_test PASSED in 33.2s //tensorflow/compiler/tests:momentum_test_cpu PASSED in 13.4s //tensorflow/compiler/tests:nary_ops_test_cpu PASSED in 12.3s //tensorflow/compiler/tests:nary_ops_test_cpu_mlir_bridge_test PASSED in 12.5s //tensorflow/compiler/tests:nullary_ops_test_cpu PASSED in 12.0s //tensorflow/compiler/tests:nullary_ops_test_cpu_mlir_bridge_test PASSED in 24.9s //tensorflow/compiler/tests:placeholder_test_cpu PASSED in 13.0s //tensorflow/compiler/tests:placeholder_test_cpu_mlir_bridge_test PASSED in 13.8s //tensorflow/compiler/tests:proximal_adagrad_test_cpu PASSED in 14.7s //tensorflow/compiler/tests:proximal_gradient_descent_test_cpu PASSED in 10.3s //tensorflow/compiler/tests:quantized_ops_test_cpu PASSED in 11.0s //tensorflow/compiler/tests:reduce_window_test_cpu PASSED in 12.9s //tensorflow/compiler/tests:reduce_window_test_cpu_mlir_bridge_test PASSED in 13.8s //tensorflow/compiler/tests:reshape_op_test_cpu PASSED in 11.9s //tensorflow/compiler/tests:reshape_op_test_cpu_mlir_bridge_test PASSED in 43.6s //tensorflow/compiler/tests:reverse_ops_test_cpu PASSED in 15.0s //tensorflow/compiler/tests:reverse_ops_test_cpu_mlir_bridge_test PASSED in 14.3s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu PASSED in 11.6s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu_mlir_bridge_test PASSED in 14.2s //tensorflow/compiler/tests:rmsprop_test_cpu PASSED in 16.7s //tensorflow/compiler/tests:scatter_nd_op_test_cpu PASSED in 26.7s //tensorflow/compiler/tests:scatter_nd_op_test_cpu_mlir_bridge_test PASSED in 32.1s //tensorflow/compiler/tests:searchsorted_op_test_cpu PASSED in 10.5s //tensorflow/compiler/tests:searchsorted_op_test_cpu_mlir_bridge_test PASSED in 14.6s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu PASSED in 25.7s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu_mlir_bridge_test PASSED in 31.4s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu PASSED in 19.3s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu_mlir_bridge_test PASSED in 21.7s //tensorflow/compiler/tests:slice_ops_test_cpu PASSED in 27.4s //tensorflow/compiler/tests:slice_ops_test_cpu_mlir_bridge_test PASSED in 24.2s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu PASSED in 12.9s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu_mlir_bridge_test PASSED in 11.8s //tensorflow/compiler/tests:stack_ops_test_cpu PASSED in 13.2s //tensorflow/compiler/tests:tensor_float_32_test_cpu PASSED in 15.5s //tensorflow/compiler/tests:tensor_float_32_test_cpu_mlir_bridge_test PASSED in 14.7s //tensorflow/compiler/tests:tensor_list_ops_test_cpu PASSED in 15.1s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu PASSED in 18.6s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu_mlir_bridge_test PASSED in 18.4s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu PASSED in 16.8s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu_mlir_bridge_test PASSED in 17.9s //tensorflow/compiler/tests:unique_ops_test_cpu PASSED in 10.7s //tensorflow/compiler/tests:variable_ops_test_cpu PASSED in 40.2s //tensorflow/compiler/tests:variable_ops_test_cpu_mlir_bridge_test PASSED in 23.0s //tensorflow/compiler/tests:where_op_test_cpu PASSED in 11.2s //tensorflow/compiler/tests:while_test_cpu PASSED in 12.7s //tensorflow/compiler/tests:xla_call_module_no_platform_check_test_cpu PASSED in 13.4s //tensorflow/compiler/tests:xla_call_module_no_shape_assertions_check_test_cpu PASSED in 13.7s //tensorflow/compiler/tests:xla_call_module_test_cpu PASSED in 16.1s //tensorflow/compiler/tests:xla_custom_call_ops_test_cpu PASSED in 11.0s //tensorflow/compiler/tests:xla_device_gpu_test_cpu PASSED in 10.7s //tensorflow/compiler/tests:xla_device_test_cpu PASSED in 22.2s //tensorflow/compiler/tests:xla_device_test_cpu_mlir_bridge_test PASSED in 15.5s //tensorflow/compiler/tests:xla_ops_test_cpu PASSED in 36.3s //tensorflow/compiler/tests:xla_ops_test_cpu_mlir_bridge_test PASSED in 45.9s //tensorflow/compiler/tests:xla_test_test PASSED in 9.8s //tensorflow/compiler/tf2xla:const_analysis_test PASSED in 5.7s //tensorflow/compiler/tf2xla:cpu_function_runtime_test PASSED in 0.1s //tensorflow/compiler/tf2xla:functionalize_cond_test PASSED in 0.8s //tensorflow/compiler/tf2xla:functionalize_control_flow_test PASSED in 0.8s //tensorflow/compiler/tf2xla:fused_batchnorm_reserve_space_test_cpu PASSED in 22.8s //tensorflow/compiler/tf2xla:graph_compiler_test PASSED in 5.2s //tensorflow/compiler/tf2xla:literal_util_test PASSED in 0.5s //tensorflow/compiler/tf2xla:resource_operation_table_test PASSED in 7.6s //tensorflow/compiler/tf2xla:resource_util_test_cpu PASSED in 1.3s //tensorflow/compiler/tf2xla:sharding_util_test PASSED in 0.7s //tensorflow/compiler/tf2xla:tf2xla_opset_test PASSED in 8.2s //tensorflow/compiler/tf2xla:tf2xla_test PASSED in 16.5s //tensorflow/compiler/tf2xla:tf2xla_util_test PASSED in 1.2s //tensorflow/compiler/tf2xla:xla_compiler_test PASSED in 17.6s //tensorflow/compiler/tf2xla:xla_jit_compiled_cpu_function_test PASSED in 14.5s //tensorflow/compiler/tf2xla:xla_op_registry_test PASSED in 5.9s //tensorflow/compiler/tf2xla/kernels:rng_converter_utils_test PASSED in 2.4s //tensorflow/compiler/xla:array2d_test PASSED in 0.1s //tensorflow/compiler/xla:array3d_test PASSED in 0.6s //tensorflow/compiler/xla:array4d_test PASSED in 0.5s //tensorflow/compiler/xla:array_test PASSED in 0.4s //tensorflow/compiler/xla:bit_cast_test PASSED in 0.2s //tensorflow/compiler/xla:comparison_util_test PASSED in 0.8s //tensorflow/compiler/xla:debug_options_parsers_test PASSED in 0.1s //tensorflow/compiler/xla:index_util_test PASSED in 0.3s //tensorflow/compiler/xla:iterator_util_test PASSED in 0.1s //tensorflow/compiler/xla:layout_test PASSED in 0.1s //tensorflow/compiler/xla:layout_util_test PASSED in 0.7s //tensorflow/compiler/xla:literal_test PASSED in 0.4s //tensorflow/compiler/xla:parse_flags_from_env_test PASSED in 0.8s //tensorflow/compiler/xla:permutation_util_test PASSED in 0.1s //tensorflow/compiler/xla:primitive_util_test PASSED in 0.2s //tensorflow/compiler/xla:refcounting_hash_map_test PASSED in 0.1s //tensorflow/compiler/xla:reference_util_test PASSED in 0.3s //tensorflow/compiler/xla:shape_test PASSED in 0.2s //tensorflow/compiler/xla:shape_tree_test PASSED in 0.2s //tensorflow/compiler/xla:shape_util_test PASSED in 2.8s //tensorflow/compiler/xla:status_macros_test PASSED in 0.3s //tensorflow/compiler/xla:text_literal_reader_test PASSED in 0.2s //tensorflow/compiler/xla:text_literal_writer_test PASSED in 0.2s //tensorflow/compiler/xla:types_test PASSED in 0.2s //tensorflow/compiler/xla:util_test PASSED in 0.3s //tensorflow/compiler/xla:window_util_test PASSED in 0.2s //tensorflow/compiler/xla/client:padding_test PASSED in 0.1s //tensorflow/compiler/xla/client:xla_builder_test PASSED in 0.4s //tensorflow/compiler/xla/client/lib:arithmetic_test_cpu PASSED in 8.2s //tensorflow/compiler/xla/client/lib:comparators_test_cpu PASSED in 7.4s //tensorflow/compiler/xla/client/lib:constants_test_cpu PASSED in 7.8s //tensorflow/compiler/xla/client/lib:logdet_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/client/lib:math_test_cpu PASSED in 12.2s //tensorflow/compiler/xla/client/lib:matrix_test_cpu PASSED in 10.5s //tensorflow/compiler/xla/client/lib:pooling_test_cpu PASSED in 8.3s //tensorflow/compiler/xla/client/lib:qr_test_cpu PASSED in 11.1s //tensorflow/compiler/xla/client/lib:slicing_test_cpu PASSED in 12.8s //tensorflow/compiler/xla/client/lib:sorting_test_cpu PASSED in 8.6s //tensorflow/compiler/xla/examples/axpy:stablehlo_compile_test PASSED in 11.5s //tensorflow/compiler/xla/experiments/sm_bandwidth_benchmark:sm_bw_test PASSED in 0.1s //tensorflow/compiler/xla/hlo/evaluator:hlo_evaluator_test PASSED in 18.9s //tensorflow/compiler/xla/hlo/experimental/auto_sharding:auto_sharding_solver_test PASSED in 1.0s //tensorflow/compiler/xla/hlo/experimental/auto_sharding:auto_sharding_test PASSED in 3.5s //tensorflow/compiler/xla/hlo/transforms:hlo_constant_splitter_test PASSED in 1.0s //tensorflow/compiler/xla/hlo/utils:hlo_live_range_test PASSED in 0.9s //tensorflow/compiler/xla/hlo/utils:hlo_matchers_test PASSED in 0.9s //tensorflow/compiler/xla/hlo/utils:hlo_sharding_util_test PASSED in 0.3s //tensorflow/compiler/xla/mlir/runtime/transforms:calling_convention_test PASSED in 0.3s //tensorflow/compiler/xla/mlir/runtime/transforms:type_converter_test PASSED in 0.3s //tensorflow/compiler/xla/mlir/tools/mlir_replay/public:execution_trace_utils_test PASSED in 0.2s //tensorflow/compiler/xla/mlir/utils:error_util_test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/bufferization/hlo_one_shot_bufferize.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_hlo_broadcasts.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_hlo_no_broadcasts.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_mhlo.mlir.test PASSED in 4.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/sparse_chlo_legalize_to_linalg.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/analysis.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/buffer_reuse.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/convert_deallocation_ops_to_llvm.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocate.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocate_invalid.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_simplification.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_to_scf.mlir.test PASSED in 1.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/split_alloc_tensors.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/add_debug_info.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/bufferization.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/collapse-shape.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/collect_stats.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/compose_extract_insert_slice.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/batch_matmul.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/conv_2d_nhwc_hwcf.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/dot.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/duplicate_fusions.mlir.test PASSED in 14.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fibonacci.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fusion_outlining.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fusion_planning_for_cpu.mlir.test PASSED in 1.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/inline_fusion_clusters.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_bcast_map.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_matmul.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_reduce_map.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_reshape_map.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/matmul.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_1d.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_1d_map.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_2d.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_window.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reverse.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/scatter.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/sort.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/transpose.mlir.test PASSED in 1.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/greedy_fusion.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/lower_vectors.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/nested_tiling_softmax.mlir.test PASSED in 12.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/ops.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/optimize_linalg_ops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/rewrite_forall_to_for.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/simplify_dead_copy.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/tile_by_one.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/tiling_softmax.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/vectorize_copy.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/vectorize_for_cpu.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-select-and-scatter.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-affine.mlir.test PASSED in 2.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-gpu.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-parallel-loops.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-tensor-op.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/ops.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo_gpu/lhlo_gpu_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/attrs.mlir.test PASSED in 1.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/broadcast_propagation.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/bitcast.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/canonicalize.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/concatenate.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/convert.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/convolution.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/custom_call.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/folder_limit.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reduce.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reshape.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reverse.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/scatter.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/transpose.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/tuple.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/while.mlir.test PASSED in 1.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/constraint_fusion.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/convert_to_signless.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/expand_hlo_tuples.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/expand_ops_simplifier.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/group_reduction_dimensions.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-collapse-elementwise-map.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-einsum-to-dot-general.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-gather-to-torch-index-select.mlir.test PASSED in 2.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-rng-to-linalg.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-shape-ops-to-standard.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-sort.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-arithmetic.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo-only-dynamic.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo-unranked.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-linalg.mlir.test PASSED in 3.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-memref-unranked.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-memref.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-stablehlo-experimental.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-stablehlo.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-torch-index-select-to-gather.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/inlining.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/invalid.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-control-flow.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-hlo-shape-computations.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-mhlo-to-thlo.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-to-std.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/lower-complex.mlir.test PASSED in 2.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/lower-general-dot.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/materialize-broadcasts.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/merge_assuming_ops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_bytecode_customizations.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_dot.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_gather.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_reduction.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_scatter.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_flatten_tuple.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_infer_shape_type_methods.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_ops_prettyprint.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_reduce_pretty_print.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/ops.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/optimize-hlo.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/prepare-for-export.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/reify-result-types.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/restrict_max_rank.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/shape_legalize_to_hlo.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/shape_reification.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sink-constants-to-control-flow.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_gendot_lower.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_lower.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_rewriting.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_transpose.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/stablehlo-legalize-to-hlo.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/symbolic-shape-optimization.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/unfuse_batch_norm.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_bounds.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_conv_op.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_reduce_op.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_reduce_window_op.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_scatter_op.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_select_and_scatter_op.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_while_op.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/while_prettyprint.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/bufferize.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/canonicalize.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/invalid.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/legalize_sort.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/ops.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/tiling.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:alloc_to_arg.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:assuming-structural-propagation.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:buffer_packing.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:bufferize.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:bufferize_one_shot.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:collapse_parallel_loops_to_1d_pass.mlir.test PASSED in 2.0s //tensorflow/compiler/xla/mlir_hlo/tests:detensorize_scf_ops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:index_type_llvm_lowering.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:legalize-trigonometric-to-approximation.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:lower_index_cast.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:propagate_static_shapes.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:rank-specialization.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:scalarization.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:shape-component-analysis.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:shape_simplification.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:test_userange.mlir.test PASSED in 2.0s //tensorflow/compiler/xla/mlir_hlo/tests:tile_loops.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:unbufferize.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:unroll-loops.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tools/mlir_interpreter/framework/tests:interpreter_value_test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tools/mlir_interpreter/framework/tests:tensor_or_memref_test PASSED in 0.1s //tensorflow/compiler/xla/pjrt:host_callback_test PASSED in 0.3s //tensorflow/compiler/xla/pjrt:lru_cache_test PASSED in 0.1s //tensorflow/compiler/xla/pjrt:pjrt_api_test PASSED in 0.3s //tensorflow/compiler/xla/pjrt:pjrt_client_test_cpu PASSED in 8.3s //tensorflow/compiler/xla/pjrt:pjrt_compiler_test PASSED in 0.3s //tensorflow/compiler/xla/pjrt:pjrt_executable_test PASSED in 1.0s //tensorflow/compiler/xla/pjrt:pjrt_stream_executor_client_test PASSED in 6.2s //tensorflow/compiler/xla/pjrt:semaphore_test PASSED in 0.3s //tensorflow/compiler/xla/pjrt:tf_pjrt_client_test PASSED in 7.1s //tensorflow/compiler/xla/pjrt:tfrt_cpu_pjrt_client_test PASSED in 7.6s //tensorflow/compiler/xla/pjrt:tracked_device_buffer_test PASSED in 6.3s //tensorflow/compiler/xla/pjrt:tracked_tfrt_cpu_device_buffer_test PASSED in 0.3s //tensorflow/compiler/xla/pjrt:transpose_test PASSED in 47.6s //tensorflow/compiler/xla/pjrt/c:pjrt_c_api_cpu_test PASSED in 10.0s //tensorflow/compiler/xla/pjrt/c:pjrt_c_api_helpers_test PASSED in 1.6s //tensorflow/compiler/xla/pjrt/distributed:client_server_test PASSED in 61.1s //tensorflow/compiler/xla/pjrt/distributed:topology_util_test PASSED in 0.3s //tensorflow/compiler/xla/python:outfeed_receiver_test_cpu PASSED in 5.7s //tensorflow/compiler/xla/python:xplane_to_profile_instructions_test PASSED in 0.4s //tensorflow/compiler/xla/python/ifrt:array_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:array_test_no_impl PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:client_test_no_impl PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:future_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:index_domain_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:index_test PASSED in 0.7s //tensorflow/compiler/xla/python/ifrt:serdes_test PASSED in 15.0s //tensorflow/compiler/xla/python/ifrt:shape_test PASSED in 0.7s //tensorflow/compiler/xla/python/ifrt:sharding_serdes_test PASSED in 0.6s //tensorflow/compiler/xla/python/ifrt:sharding_test PASSED in 0.4s //tensorflow/compiler/xla/python/ifrt:tuple_test_no_impl PASSED in 0.2s //tensorflow/compiler/xla/python/ifrt/ir/tests:executable_test_no_impl PASSED in 7.7s //tensorflow/compiler/xla/python/ifrt/support:sharding_param_to_op_sharding_test PASSED in 0.3s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_array_impl_test_tfrt_cpu PASSED in 15.4s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_client_impl_test_tfrt_cpu PASSED in 7.8s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_executable_impl_test_tfrt_cpu PASSED in 8.5s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_tuple_impl_test_tfrt_cpu PASSED in 8.3s //tensorflow/compiler/xla/python/pjrt_ifrt:xla_executable_test_no_impl PASSED in 3.3s //tensorflow/compiler/xla/python/pjrt_ifrt:xla_program_serdes_test PASSED in 2.0s //tensorflow/compiler/xla/python/pjrt_ifrt:xla_sharding_serdes_test PASSED in 0.4s //tensorflow/compiler/xla/python/pjrt_ifrt:xla_sharding_test PASSED in 7.8s //tensorflow/compiler/xla/python_api:xla_literal_test PASSED in 1.8s //tensorflow/compiler/xla/python_api:xla_shape_test PASSED in 2.0s //tensorflow/compiler/xla/rpc:grpc_client_test PASSED in 2.7s //tensorflow/compiler/xla/runtime:arguments_test PASSED in 0.4s //tensorflow/compiler/xla/runtime:async_runtime_test PASSED in 1.0s //tensorflow/compiler/xla/runtime:custom_call_test PASSED in 1.9s //tensorflow/compiler/xla/runtime:diagnostics_test PASSED in 0.7s //tensorflow/compiler/xla/runtime:executable_test PASSED in 2.9s //tensorflow/compiler/xla/runtime:ffi_test PASSED in 1.6s //tensorflow/compiler/xla/runtime:map_by_type_test PASSED in 0.3s //tensorflow/compiler/xla/runtime:module_test PASSED in 0.5s //tensorflow/compiler/xla/runtime:results_test PASSED in 0.3s //tensorflow/compiler/xla/runtime:state_test PASSED in 0.1s //tensorflow/compiler/xla/runtime:symbolic_shape_test PASSED in 0.8s //tensorflow/compiler/xla/runtime:type_id_test PASSED in 0.6s //tensorflow/compiler/xla/service:algebraic_simplifier_overflow_test_cpu PASSED in 8.0s //tensorflow/compiler/xla/service:algebraic_simplifier_test PASSED in 2.3s //tensorflow/compiler/xla/service:all_gather_broadcast_reorder_test PASSED in 0.7s //tensorflow/compiler/xla/service:all_gather_combiner_test PASSED in 2.2s //tensorflow/compiler/xla/service:all_gather_decomposer_test PASSED in 1.0s //tensorflow/compiler/xla/service:all_reduce_combiner_test PASSED in 0.8s //tensorflow/compiler/xla/service:all_reduce_contiguous_test PASSED in 1.0s //tensorflow/compiler/xla/service:all_reduce_folder_test PASSED in 0.8s //tensorflow/compiler/xla/service:all_reduce_promotion_test PASSED in 0.7s //tensorflow/compiler/xla/service:all_reduce_reassociate_test PASSED in 1.2s //tensorflow/compiler/xla/service:all_reduce_simplifier_test PASSED in 0.8s //tensorflow/compiler/xla/service:ar_crs_combiner_test PASSED in 0.8s //tensorflow/compiler/xla/service:async_collective_creator_test PASSED in 0.8s //tensorflow/compiler/xla/service:async_op_canonicalizer_test PASSED in 1.3s //tensorflow/compiler/xla/service:batch_dot_simplification_test PASSED in 0.6s //tensorflow/compiler/xla/service:batchnorm_expander_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/service:bfloat16_conversion_folding_test PASSED in 0.8s //tensorflow/compiler/xla/service:bfloat16_propagation_test PASSED in 1.2s //tensorflow/compiler/xla/service:bitcast_dtypes_expander_test PASSED in 0.8s //tensorflow/compiler/xla/service:broadcast_canonicalizer_test PASSED in 1.3s //tensorflow/compiler/xla/service:buffer_assignment_test PASSED in 10.5s //tensorflow/compiler/xla/service:call_graph_test PASSED in 0.8s //tensorflow/compiler/xla/service:call_inliner_test PASSED in 0.7s //tensorflow/compiler/xla/service:change_op_data_type_test PASSED in 1.5s //tensorflow/compiler/xla/service:collective_ops_utils_test PASSED in 0.7s //tensorflow/compiler/xla/service:collective_permute_decomposer_test PASSED in 1.8s //tensorflow/compiler/xla/service:collective_pipeliner_test PASSED in 0.7s //tensorflow/compiler/xla/service:collective_transformation_reorderer_test PASSED in 1.4s //tensorflow/compiler/xla/service:collectives_schedule_linearizer_test PASSED in 2.6s //tensorflow/compiler/xla/service:compilation_environments_test PASSED in 0.2s //tensorflow/compiler/xla/service:conditional_canonicalizer_test PASSED in 1.0s //tensorflow/compiler/xla/service:conditional_code_motion_test PASSED in 1.1s //tensorflow/compiler/xla/service:conditional_simplifier_test PASSED in 1.2s //tensorflow/compiler/xla/service:conditional_to_select_test PASSED in 0.7s //tensorflow/compiler/xla/service:constant_value_test PASSED in 0.2s //tensorflow/compiler/xla/service:convert_async_collectives_to_sync_test PASSED in 1.0s //tensorflow/compiler/xla/service:convert_mover_test PASSED in 1.3s //tensorflow/compiler/xla/service:convert_operand_folding_test PASSED in 0.7s //tensorflow/compiler/xla/service:convolution_4d_expander_test PASSED in 0.7s //tensorflow/compiler/xla/service:convolution_group_converter_test PASSED in 1.1s //tensorflow/compiler/xla/service:convolution_pred_expander_test PASSED in 1.0s //tensorflow/compiler/xla/service:copy_insertion_test PASSED in 0.8s //tensorflow/compiler/xla/service:custom_call_status_test PASSED in 0.1s //tensorflow/compiler/xla/service:defuser_test PASSED in 0.7s //tensorflow/compiler/xla/service:despecializer_test PASSED in 1.7s //tensorflow/compiler/xla/service:dfs_hlo_visitor_with_default_test PASSED in 0.6s //tensorflow/compiler/xla/service:dot_decomposer_test PASSED in 0.8s //tensorflow/compiler/xla/service:dot_dimension_merger_test PASSED in 1.4s //tensorflow/compiler/xla/service:dot_merger_test PASSED in 0.9s //tensorflow/compiler/xla/service:dynamic_dimension_inference_test PASSED in 1.0s //tensorflow/compiler/xla/service:dynamic_dimension_simplifier_test PASSED in 1.0s //tensorflow/compiler/xla/service:dynamic_index_splitter_test PASSED in 1.4s //tensorflow/compiler/xla/service:dynamic_padder_test_cpu PASSED in 14.2s //tensorflow/compiler/xla/service:dynamic_parameter_binding_test PASSED in 1.0s //tensorflow/compiler/xla/service:dynamic_update_slice_test_cpu PASSED in 10.0s //tensorflow/compiler/xla/service:elemental_ir_emitter_test_cpu PASSED in 14.1s //tensorflow/compiler/xla/service:flatten_call_graph_test PASSED in 0.9s //tensorflow/compiler/xla/service:float_normalization_test PASSED in 1.0s //tensorflow/compiler/xla/service:fusion_node_indexing_evaluation_test PASSED in 0.9s //tensorflow/compiler/xla/service:gather_expander_test PASSED in 0.8s //tensorflow/compiler/xla/service:gather_simplifier_test PASSED in 0.9s //tensorflow/compiler/xla/service:heap_simulator_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_alias_analysis_test PASSED in 1.5s //tensorflow/compiler/xla/service:hlo_casting_utils_test PASSED in 9.1s //tensorflow/compiler/xla/service:hlo_computation_deduplicator_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_computation_test PASSED in 2.2s //tensorflow/compiler/xla/service:hlo_constant_folding_test PASSED in 5.4s //tensorflow/compiler/xla/service:hlo_cost_analysis_test PASSED in 7.8s //tensorflow/compiler/xla/service:hlo_creation_utils_test PASSED in 3.3s //tensorflow/compiler/xla/service:hlo_cse_test PASSED in 6.7s //tensorflow/compiler/xla/service:hlo_dataflow_analysis_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_dce_test PASSED in 0.6s //tensorflow/compiler/xla/service:hlo_domain_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_element_type_converter_test PASSED in 2.1s //tensorflow/compiler/xla/service:hlo_execution_profile_test PASSED in 5.8s //tensorflow/compiler/xla/service:hlo_graph_dumper_test PASSED in 1.0s //tensorflow/compiler/xla/service:hlo_input_output_alias_config_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_instruction_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_liveness_analysis_test PASSED in 1.8s //tensorflow/compiler/xla/service:hlo_memory_scheduler_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_module_dce_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_module_metadata_test PASSED in 0.1s //tensorflow/compiler/xla/service:hlo_module_test PASSED in 1.8s //tensorflow/compiler/xla/service:hlo_opcode_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_ordering_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_parser_test PASSED in 0.3s //tensorflow/compiler/xla/service:hlo_pass_pipeline_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_phi_graph_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_proto_util_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_reachability_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_rematerialization_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_rematerialization_test_utils_test PASSED in 1.0s //tensorflow/compiler/xla/service:hlo_replication_analysis_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_schedule_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_sharding_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_value_semantics_analysis_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_verifier_test PASSED in 1.2s //tensorflow/compiler/xla/service:indexed_array_analysis_test PASSED in 0.8s //tensorflow/compiler/xla/service:instruction_fusion_test PASSED in 0.8s //tensorflow/compiler/xla/service:latency_hiding_scheduler_test PASSED in 0.8s //tensorflow/compiler/xla/service:layout_assignment_test PASSED in 5.2s //tensorflow/compiler/xla/service:layout_normalization_test PASSED in 1.3s //tensorflow/compiler/xla/service:logistic_expander_test PASSED in 0.7s //tensorflow/compiler/xla/service:loop_schedule_linearizer_test PASSED in 1.0s //tensorflow/compiler/xla/service:map_inliner_test PASSED in 0.9s //tensorflow/compiler/xla/service:mapped_ptr_container_sorter_test PASSED in 0.1s //tensorflow/compiler/xla/service:memory_space_assignment_best_fit_repacker_test PASSED in 0.3s //tensorflow/compiler/xla/service:memory_space_assignment_test PASSED in 9.2s //tensorflow/compiler/xla/service:memory_space_propagation_test PASSED in 0.7s //tensorflow/compiler/xla/service:name_uniquer_test PASSED in 0.1s //tensorflow/compiler/xla/service:operand_upcaster_test PASSED in 1.0s //tensorflow/compiler/xla/service:optimize_input_output_buffer_alias_test PASSED in 0.8s //tensorflow/compiler/xla/service:pattern_matcher_gmock_test PASSED in 0.2s //tensorflow/compiler/xla/service:pattern_matcher_test PASSED in 1.1s //tensorflow/compiler/xla/service:profile_guided_latency_estimator_test PASSED in 17.9s //tensorflow/compiler/xla/service:real_imag_expander_test PASSED in 2.3s //tensorflow/compiler/xla/service:reduce_decomposer_test PASSED in 0.7s //tensorflow/compiler/xla/service:reduce_scatter_combiner_test PASSED in 0.7s //tensorflow/compiler/xla/service:reduce_scatter_decomposer_test PASSED in 0.7s //tensorflow/compiler/xla/service:reduce_scatter_reassociate_test PASSED in 2.0s //tensorflow/compiler/xla/service:reshape_decomposer_test PASSED in 0.9s //tensorflow/compiler/xla/service:reshape_mover_test PASSED in 2.8s //tensorflow/compiler/xla/service:result_caster_test PASSED in 0.7s //tensorflow/compiler/xla/service:root_instruction_sinker_test PASSED in 1.6s //tensorflow/compiler/xla/service:scatter_expander_test PASSED in 6.2s //tensorflow/compiler/xla/service:scatter_simplifier_test PASSED in 0.8s //tensorflow/compiler/xla/service:select_and_scatter_expander_test PASSED in 0.8s //tensorflow/compiler/xla/service:shape_inference_test PASSED in 0.4s //tensorflow/compiler/xla/service:shaped_buffer_test PASSED in 6.6s //tensorflow/compiler/xla/service:sharding_propagation_test PASSED in 2.4s //tensorflow/compiler/xla/service:sharding_remover_test PASSED in 1.0s //tensorflow/compiler/xla/service:simplify_fp_conversions_test PASSED in 1.7s //tensorflow/compiler/xla/service:slice_sinker_test PASSED in 13.6s //tensorflow/compiler/xla/service:sort_simplifier_test PASSED in 1.7s //tensorflow/compiler/xla/service:space_to_batch_converter_test PASSED in 0.7s //tensorflow/compiler/xla/service:stable_sort_expander_test PASSED in 2.6s //tensorflow/compiler/xla/service:stochastic_convert_decomposer_test PASSED in 0.9s //tensorflow/compiler/xla/service:stream_pool_test PASSED in 0.3s //tensorflow/compiler/xla/service:topk_rewriter_test PASSED in 3.1s //tensorflow/compiler/xla/service:transpose_folding_test PASSED in 1.2s //tensorflow/compiler/xla/service:tuple_points_to_analysis_test PASSED in 1.3s //tensorflow/compiler/xla/service:tuple_simplifier_test PASSED in 1.2s //tensorflow/compiler/xla/service:tuple_util_test PASSED in 1.2s //tensorflow/compiler/xla/service:value_range_test PASSED in 0.9s //tensorflow/compiler/xla/service:while_loop_all_reduce_code_motion_test PASSED in 1.8s //tensorflow/compiler/xla/service:while_loop_analysis_test PASSED in 1.0s //tensorflow/compiler/xla/service:while_loop_concat_code_motion_test PASSED in 2.4s //tensorflow/compiler/xla/service:while_loop_constant_sinking_test PASSED in 1.2s //tensorflow/compiler/xla/service:while_loop_expensive_invariant_code_motion_test PASSED in 1.0s //tensorflow/compiler/xla/service:while_loop_invariant_code_motion_test PASSED in 0.7s //tensorflow/compiler/xla/service:while_loop_simplifier_test PASSED in 0.7s //tensorflow/compiler/xla/service:while_loop_trip_count_annotator_test PASSED in 1.1s //tensorflow/compiler/xla/service:while_util_test PASSED in 0.8s //tensorflow/compiler/xla/service:xla_aot_compile_stablehlo_cpu_test PASSED in 9.4s //tensorflow/compiler/xla/service:xla_debug_info_manager_test PASSED in 2.0s //tensorflow/compiler/xla/service:zero_sized_hlo_elimination_test PASSED in 0.9s //tensorflow/compiler/xla/service/cpu:conv_canonicalization_test PASSED in 1.5s //tensorflow/compiler/xla/service/cpu:cpu_eigen_tensor_alignment_test PASSED in 0.9s //tensorflow/compiler/xla/service/cpu:cpu_instruction_fusion_test PASSED in 0.9s //tensorflow/compiler/xla/service/cpu:cpu_layout_assignment_test PASSED in 1.9s //tensorflow/compiler/xla/service/cpu:ir_emission_utils_test PASSED in 1.2s //tensorflow/compiler/xla/service/cpu:parallel_task_assignment_test PASSED in 3.7s //tensorflow/compiler/xla/service/cpu:runtime_fft_test PASSED in 0.3s //tensorflow/compiler/xla/service/cpu:shape_partition_test PASSED in 1.2s //tensorflow/compiler/xla/service/cpu:xfeed_manager_test PASSED in 1.2s //tensorflow/compiler/xla/service/cpu/tests:cpu_bytesizeof_test PASSED in 0.6s //tensorflow/compiler/xla/service/cpu/tests:cpu_dyn_shape_test PASSED in 7.4s //tensorflow/compiler/xla/service/cpu/tests:cpu_eigen_dot_operation_test PASSED in 7.4s //tensorflow/compiler/xla/service/cpu/tests:cpu_external_constants_test PASSED in 21.4s //tensorflow/compiler/xla/service/cpu/tests:cpu_fusion_test PASSED in 7.6s //tensorflow/compiler/xla/service/cpu/tests:cpu_infeed_test PASSED in 6.4s //tensorflow/compiler/xla/service/cpu/tests:cpu_intrinsic_test PASSED in 8.0s //tensorflow/compiler/xla/service/cpu/tests:cpu_key_value_sort_test PASSED in 7.4s //tensorflow/compiler/xla/service/cpu/tests:cpu_literal_caching_test PASSED in 7.4s //tensorflow/compiler/xla/service/cpu/tests:cpu_noalias_test PASSED in 8.6s //tensorflow/compiler/xla/service/cpu/tests:cpu_outfeed_test PASSED in 5.6s //tensorflow/compiler/xla/service/cpu/tests:cpu_profiling_test PASSED in 9.6s //tensorflow/compiler/xla/service/cpu/tests:cpu_spmd_compile_test PASSED in 6.9s //tensorflow/compiler/xla/service/cpu/tests:cpu_topk_test PASSED in 9.0s //tensorflow/compiler/xla/service/cpu/tests:cpu_vectorization_test PASSED in 8.2s //tensorflow/compiler/xla/service/cpu/tests:cpu_while_test PASSED in 9.4s //tensorflow/compiler/xla/service/cpu/tests:tree_reduction_rewriter_test PASSED in 8.5s //tensorflow/compiler/xla/service/gpu:alias_passthrough_params_test PASSED in 1.0s //tensorflow/compiler/xla/service/gpu:all_reduce_blueconnect_test PASSED in 1.3s //tensorflow/compiler/xla/service/gpu:autotuner_util_test PASSED in 0.2s //tensorflow/compiler/xla/service/gpu:backend_configs_test PASSED in 0.6s //tensorflow/compiler/xla/service/gpu:copy_fusion_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu:cublas_pad_for_gemms_test PASSED in 1.7s //tensorflow/compiler/xla/service/gpu:cudnn_pad_for_convolutions_test PASSED in 2.1s //tensorflow/compiler/xla/service/gpu:cudnn_simplify_padding_test PASSED in 1.2s //tensorflow/compiler/xla/service/gpu:cudnn_support_utils_test PASSED in 1.3s //tensorflow/compiler/xla/service/gpu:cudnn_vectorize_convolutions_test PASSED in 1.0s //tensorflow/compiler/xla/service/gpu:gemm_rewriter_triton_test PASSED in 5.9s //tensorflow/compiler/xla/service/gpu:gpu_async_collective_annotator_test PASSED in 2.2s //tensorflow/compiler/xla/service/gpu:gpu_conv_padding_legalization_test PASSED in 1.0s //tensorflow/compiler/xla/service/gpu:gpu_conv_rewriter_test PASSED in 0.8s //tensorflow/compiler/xla/service/gpu:gpu_convert_async_collectives_to_sync_test PASSED in 0.7s //tensorflow/compiler/xla/service/gpu:gpu_fusible_test PASSED in 3.5s //tensorflow/compiler/xla/service/gpu:gpu_hlo_cost_analysis_test PASSED in 1.0s //tensorflow/compiler/xla/service/gpu:gpu_performance_model_test PASSED in 4.4s //tensorflow/compiler/xla/service/gpu:gpu_sanitize_constant_names_test PASSED in 1.0s //tensorflow/compiler/xla/service/gpu:hlo_algorithm_denylist_test PASSED in 0.2s //tensorflow/compiler/xla/service/gpu:hlo_fusion_stats_test PASSED in 1.1s //tensorflow/compiler/xla/service/gpu:instruction_fusion_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu:ir_emission_utils_test PASSED in 1.1s //tensorflow/compiler/xla/service/gpu:matmul_utils_test PASSED in 1.9s //tensorflow/compiler/xla/service/gpu:move_copy_to_users_test PASSED in 1.1s //tensorflow/compiler/xla/service/gpu:multi_output_fusion_test PASSED in 1.7s //tensorflow/compiler/xla/service/gpu:non_atomically_upgradeable_rw_lock_test PASSED in 0.2s //tensorflow/compiler/xla/service/gpu:priority_fusion_test PASSED in 1.2s //tensorflow/compiler/xla/service/gpu:reduction_splitter_test PASSED in 1.5s //tensorflow/compiler/xla/service/gpu:scatter_slice_simplifier_test PASSED in 1.8s //tensorflow/compiler/xla/service/gpu:softmax_rewriter_triton_test PASSED in 4.4s //tensorflow/compiler/xla/service/gpu:target_util_test PASSED in 0.5s //tensorflow/compiler/xla/service/gpu:topk_splitter_test PASSED in 49.3s //tensorflow/compiler/xla/service/gpu:variadic_op_splitter_test PASSED in 1.6s //tensorflow/compiler/xla/service/gpu:while_transformer_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu/llvm_gpu_backend:utils_test PASSED in 0.3s //tensorflow/compiler/xla/service/gpu/tests:gpu_reduce_scatter_creator_test PASSED in 3.3s //tensorflow/compiler/xla/service/gpu/tests:reduction_degenerate_dim_remover_test PASSED in 1.5s //tensorflow/compiler/xla/service/gpu/tests:reduction_dimension_grouper_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu/tests:tree_reduction_rewriter_test PASSED in 2.0s //tensorflow/compiler/xla/service/graphcycles:graphcycles_test PASSED in 0.7s //tensorflow/compiler/xla/service/graphcycles:ordered_set_test PASSED in 0.2s //tensorflow/compiler/xla/service/llvm_ir:alias_analysis_test PASSED in 7.8s //tensorflow/compiler/xla/service/llvm_ir:ir_array_test PASSED in 0.6s //tensorflow/compiler/xla/service/spmd:canonicalize_all_gather_for_cse_test PASSED in 0.8s //tensorflow/compiler/xla/service/spmd:collective_permute_motion_test PASSED in 0.8s //tensorflow/compiler/xla/service/spmd:partition_assignment_test PASSED in 0.7s //tensorflow/compiler/xla/service/spmd:schedule_aware_collective_ops_cse_test PASSED in 1.0s //tensorflow/compiler/xla/service/spmd:spmd_partitioner_test PASSED in 3.8s //tensorflow/compiler/xla/service/spmd:spmd_prepare_test PASSED in 0.8s //tensorflow/compiler/xla/service/spmd:stateful_rng_spmd_partitioner_test PASSED in 0.8s //tensorflow/compiler/xla/stream_executor:dnn_test PASSED in 0.2s //tensorflow/compiler/xla/stream_executor:stream_test PASSED in 0.3s //tensorflow/compiler/xla/stream_executor/host:host_stream_test PASSED in 0.1s //tensorflow/compiler/xla/stream_executor/tpu:c_api_conversions_test PASSED in 0.6s //tensorflow/compiler/xla/tests:all_reduce_test_cpu PASSED in 6.5s //tensorflow/compiler/xla/tests:axpy_simple_test_cpu PASSED in 6.6s //tensorflow/compiler/xla/tests:bad_rng_shape_validation_test_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests:binop_scaling_test_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests:bitcast_convert_test_cpu PASSED in 7.9s //tensorflow/compiler/xla/tests:broadcast_simple_test_cpu PASSED in 9.2s //tensorflow/compiler/xla/tests:broadcast_test_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests:buffer_donation_test_cpu PASSED in 9.2s //tensorflow/compiler/xla/tests:call_test_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests:check_execution_arity_test_cpu PASSED in 8.1s //tensorflow/compiler/xla/tests:cholesky_test_cpu PASSED in 14.7s //tensorflow/compiler/xla/tests:client_test_cpu PASSED in 8.2s //tensorflow/compiler/xla/tests:collective_ops_test_cpu PASSED in 50.1s //tensorflow/compiler/xla/tests:collective_pipeliner_execution_test_cpu PASSED in 12.5s //tensorflow/compiler/xla/tests:compilation_cache_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests:compute_constant_test_cpu PASSED in 6.7s //tensorflow/compiler/xla/tests:concat_test_cpu PASSED in 8.0s //tensorflow/compiler/xla/tests:constant_reduction_function_test_cpu PASSED in 6.7s //tensorflow/compiler/xla/tests:constants_test_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests:convert_test_cpu PASSED in 17.2s //tensorflow/compiler/xla/tests:copy_test_cpu PASSED in 9.5s //tensorflow/compiler/xla/tests:cpu_gpu_fusion_test_cpu PASSED in 9.9s //tensorflow/compiler/xla/tests:custom_call_test_cpu PASSED in 9.9s //tensorflow/compiler/xla/tests:deallocation_test_cpu PASSED in 8.3s //tensorflow/compiler/xla/tests:deconstruct_tuple_test_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests:deep_graph_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests:fft_test_cpu PASSED in 8.6s //tensorflow/compiler/xla/tests:float8_test_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests:floor_ceil_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests:fmax_fmin_test_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests:gather_operation_test_cpu PASSED in 18.3s //tensorflow/compiler/xla/tests:get_dimension_size_test_cpu PASSED in 11.1s //tensorflow/compiler/xla/tests:half_test_cpu PASSED in 8.1s //tensorflow/compiler/xla/tests:hlo_metadata_test PASSED in 7.5s //tensorflow/compiler/xla/tests:literal_test_util_test PASSED in 5.4s //tensorflow/compiler/xla/tests:local_client_allocation_test_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests:local_client_aot_test PASSED in 0.2s //tensorflow/compiler/xla/tests:log_test_cpu PASSED in 6.7s //tensorflow/compiler/xla/tests:map_test_cpu PASSED in 8.4s //tensorflow/compiler/xla/tests:matrix_ops_simple_test_cpu PASSED in 14.3s //tensorflow/compiler/xla/tests:multidimensional_slice_test_cpu PASSED in 8.7s //tensorflow/compiler/xla/tests:multiple_devices_on_host_test PASSED in 6.7s //tensorflow/compiler/xla/tests:multithreaded_compilation_test_cpu PASSED in 8.9s //tensorflow/compiler/xla/tests:outfeed_in_nested_computation_test_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests:pad_test_cpu PASSED in 9.2s //tensorflow/compiler/xla/tests:pred_test_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests:query_inferred_shape_test_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests:reduce_hlo_test_cpu PASSED in 9.7s //tensorflow/compiler/xla/tests:reduce_precision_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests:replay_test_cpu PASSED in 8.0s //tensorflow/compiler/xla/tests:reshape_motion_test_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests:reverse_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests:round_trip_packed_literal_test_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests:round_trip_transfer_test_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests:sample_text_test_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests:scatter_test_cpu PASSED in 10.3s //tensorflow/compiler/xla/tests:select_test_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests:test_utils_test_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests:tile_assignment_test PASSED in 0.2s //tensorflow/compiler/xla/tests:token_hlo_test_cpu PASSED in 9.0s //tensorflow/compiler/xla/tests:topk_test_cpu PASSED in 8.4s //tensorflow/compiler/xla/tests:transfer_manager_test_cpu PASSED in 17.5s //tensorflow/compiler/xla/tests:transpose_test_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests:tuple_test_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests:unary_op_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests:value_inference_test_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests:vector_ops_reduce_test_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests:vector_ops_simple_test_cpu PASSED in 8.1s //tensorflow/compiler/xla/tests:while_test_cpu PASSED in 9.0s //tensorflow/compiler/xla/tests/fuzz:rand_000000_cpu PASSED in 9.4s //tensorflow/compiler/xla/tests/fuzz:rand_000003_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests/fuzz:rand_000005_cpu PASSED in 8.6s //tensorflow/compiler/xla/tests/fuzz:rand_000006_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests/fuzz:rand_000007_cpu PASSED in 9.1s //tensorflow/compiler/xla/tests/fuzz:rand_000008_cpu PASSED in 9.0s //tensorflow/compiler/xla/tests/fuzz:rand_000009_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests/fuzz:rand_000013_cpu PASSED in 8.4s //tensorflow/compiler/xla/tests/fuzz:rand_000015_cpu PASSED in 8.7s //tensorflow/compiler/xla/tests/fuzz:rand_000016_cpu PASSED in 10.2s //tensorflow/compiler/xla/tests/fuzz:rand_000017_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests/fuzz:rand_000018_cpu PASSED in 8.7s //tensorflow/compiler/xla/tests/fuzz:rand_000019_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests/fuzz:rand_000020_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests/fuzz:rand_000022_cpu PASSED in 9.4s //tensorflow/compiler/xla/tests/fuzz:rand_000024_cpu PASSED in 8.3s //tensorflow/compiler/xla/tests/fuzz:rand_000025_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests/fuzz:rand_000026_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests/fuzz:rand_000030_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests/fuzz:rand_000031_cpu PASSED in 8.9s //tensorflow/compiler/xla/tests/fuzz:rand_000032_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests/fuzz:rand_000033_cpu PASSED in 8.7s //tensorflow/compiler/xla/tests/fuzz:rand_000034_cpu PASSED in 8.4s //tensorflow/compiler/xla/tests/fuzz:rand_000035_cpu PASSED in 8.4s //tensorflow/compiler/xla/tests/fuzz:rand_000036_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests/fuzz:rand_000039_cpu PASSED in 9.2s //tensorflow/compiler/xla/tests/fuzz:rand_000040_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests/fuzz:rand_000041_cpu PASSED in 10.0s //tensorflow/compiler/xla/tests/fuzz:rand_000043_cpu PASSED in 6.2s //tensorflow/compiler/xla/tests/fuzz:rand_000049_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests/fuzz:rand_000053_cpu PASSED in 7.9s //tensorflow/compiler/xla/tests/fuzz:rand_000056_cpu PASSED in 8.6s //tensorflow/compiler/xla/tests/fuzz:rand_000059_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests/fuzz:rand_000061_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests/fuzz:rand_000062_cpu PASSED in 9.0s //tensorflow/compiler/xla/tests/fuzz:rand_000064_cpu PASSED in 9.3s //tensorflow/compiler/xla/tests/fuzz:rand_000066_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests/fuzz:rand_000069_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests/fuzz:rand_000071_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests/fuzz:rand_000077_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests/fuzz:rand_000078_cpu PASSED in 7.9s //tensorflow/compiler/xla/tests/fuzz:rand_000079_cpu PASSED in 7.9s //tensorflow/compiler/xla/tests/fuzz:rand_000081_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests/fuzz:rand_000084_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests/fuzz:rand_000085_cpu PASSED in 9.4s //tensorflow/compiler/xla/tests/fuzz:rand_000086_cpu PASSED in 6.2s //tensorflow/compiler/xla/tests/fuzz:rand_000088_cpu PASSED in 7.9s //tensorflow/compiler/xla/tests/fuzz:rand_000089_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests/fuzz:rand_000090_cpu PASSED in 6.0s //tensorflow/compiler/xla/tests/fuzz:rand_000092_cpu PASSED in 8.8s //tensorflow/compiler/xla/tests/fuzz:rand_000094_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests/fuzz:rand_000095_cpu PASSED in 8.7s //tensorflow/compiler/xla/tools:hlo_control_flow_flattening_test PASSED in 1.1s //tensorflow/compiler/xla/tools:hlo_extractor_test PASSED in 0.7s //tensorflow/compiler/xla/tools:hlo_module_loader_test PASSED in 0.8s //tensorflow/compiler/xla/tools:hlo_slicer_test PASSED in 0.7s //tensorflow/compiler/xla/tools:interactive_graphviz_bin_test PASSED in 0.2s //tensorflow/compiler/xla/tools:run_hlo_module_bin_test PASSED in 0.3s //tensorflow/compiler/xla/tools/hlo_bisect:hlo_bisect_state_test PASSED in 1.1s //tensorflow/compiler/xla/translate/hlo_to_mhlo:hlo_utils_test PASSED in 0.9s //tensorflow/compiler/xla/translate/mhlo_to_hlo:type_to_shape_test PASSED in 1.3s //tensorflow/core:__tensorflow_core_lib_core_legacy_lib_core_all_tests PASSED in 5.9s //tensorflow/core:__tensorflow_core_lib_gtl_legacy_lib_gtl_tests PASSED in 0.6s //tensorflow/core:__tensorflow_core_lib_monitoring_cell_reader_test PASSED in 35.0s //tensorflow/core:__tensorflow_core_lib_monitoring_collection_registry_test PASSED in 0.4s //tensorflow/core:__tensorflow_core_lib_monitoring_counter_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_gauge_test PASSED in 6.1s //tensorflow/core:__tensorflow_core_lib_monitoring_metric_def_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_percentile_sampler_test PASSED in 0.4s //tensorflow/core:__tensorflow_core_lib_monitoring_sampler_test PASSED in 0.3s //tensorflow/core:__tensorflow_core_lib_monitoring_test_utils_test PASSED in 0.3s //tensorflow/core:__tensorflow_core_lib_strings_legacy_low_level_library_tests PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_wav_wav_io_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_util_mkl_util_test_srcs PASSED in 0.6s //tensorflow/core:__tensorflow_tsl_lib_core_legacy_lib_core_all_tests PASSED in 0.4s //tensorflow/core:lib_strings_ordered_code_test PASSED in 1.4s //tensorflow/core:lib_strings_proto_serialization_test PASSED in 0.2s //tensorflow/core/api_def:api_test PASSED in 4.3s //tensorflow/core/api_def:update_api_def_test PASSED in 0.1s //tensorflow/core/common_runtime:all_to_all_test_cpu PASSED in 0.7s //tensorflow/core/common_runtime:arg_ret_placement_test PASSED in 0.6s //tensorflow/core/common_runtime:buf_rendezvous_test PASSED in 1.8s //tensorflow/core/common_runtime:collective_executor_mgr_test PASSED in 0.8s //tensorflow/core/common_runtime:collective_param_resolver_local_test PASSED in 4.8s //tensorflow/core/common_runtime:collective_rma_local_test PASSED in 1.4s //tensorflow/core/common_runtime:composite_device_test PASSED in 0.9s //tensorflow/core/common_runtime:cost_measurement_registry_test PASSED in 2.2s //tensorflow/core/common_runtime:cost_util_test PASSED in 0.1s //tensorflow/core/common_runtime:device_mgr_test PASSED in 0.7s //tensorflow/core/common_runtime:device_propagation_test PASSED in 1.1s //tensorflow/core/common_runtime:device_resolver_local_test PASSED in 1.0s //tensorflow/core/common_runtime:device_set_test PASSED in 1.0s //tensorflow/core/common_runtime:direct_session_test_cpu PASSED in 2.4s //tensorflow/core/common_runtime:direct_session_with_debug_test PASSED in 2.1s //tensorflow/core/common_runtime:direct_session_with_tracking_alloc_test PASSED in 1.1s //tensorflow/core/common_runtime:dynamic_device_mgr_test PASSED in 1.5s //tensorflow/core/common_runtime:eval_const_tensor_test PASSED in 0.7s //tensorflow/core/common_runtime:executor_test PASSED in 2.0s //tensorflow/core/common_runtime:function_optimization_registration_test PASSED in 0.9s //tensorflow/core/common_runtime:function_optimization_registry_no_pass_test PASSED in 1.8s //tensorflow/core/common_runtime:function_optimization_registry_pass_failure_test PASSED in 1.4s //tensorflow/core/common_runtime:function_optimization_registry_test PASSED in 1.6s //tensorflow/core/common_runtime:function_threadpool_test PASSED in 1.0s //tensorflow/core/common_runtime:graph_constructor_test PASSED in 1.5s //tensorflow/core/common_runtime:graph_runner_test PASSED in 1.4s //tensorflow/core/common_runtime:hierarchical_tree_broadcaster_test_cpu PASSED in 4.2s //tensorflow/core/common_runtime:inline_function_utils_test PASSED in 1.9s //tensorflow/core/common_runtime:input_colocation_exemption_registry_test PASSED in 0.5s //tensorflow/core/common_runtime:int32_fulltype_test PASSED in 0.6s //tensorflow/core/common_runtime:isolate_placer_inspection_required_ops_pass_test PASSED in 0.9s //tensorflow/core/common_runtime:lower_case_op_test PASSED in 1.9s //tensorflow/core/common_runtime:lower_function_call_test PASSED in 2.2s //tensorflow/core/common_runtime:lower_functional_ops_test PASSED in 2.2s //tensorflow/core/common_runtime:lower_if_op_test PASSED in 1.9s //tensorflow/core/common_runtime:lower_while_op_test PASSED in 2.0s //tensorflow/core/common_runtime:mkl_cpu_allocator_test PASSED in 0.2s //tensorflow/core/common_runtime:mkl_threadpool_device_test PASSED in 0.1s //tensorflow/core/common_runtime:no_op_cost_measurement_test PASSED in 0.1s //tensorflow/core/common_runtime:null_request_cost_accessor_test PASSED in 0.1s //tensorflow/core/common_runtime:optimization_registry_test PASSED in 1.6s //tensorflow/core/common_runtime:optimize_cross_host_control_deps_test PASSED in 5.9s //tensorflow/core/common_runtime:optimize_function_graph_utils_test PASSED in 0.6s //tensorflow/core/common_runtime:partitioning_utils_test PASSED in 0.7s //tensorflow/core/common_runtime:pending_counts_test PASSED in 1.9s //tensorflow/core/common_runtime:permuter_test_cpu PASSED in 3.7s //tensorflow/core/common_runtime:placer_inspection_required_ops_utils_test PASSED in 1.1s //tensorflow/core/common_runtime:placer_test PASSED in 0.9s //tensorflow/core/common_runtime:process_function_library_runtime_test_cpu PASSED in 0.7s //tensorflow/core/common_runtime:process_util_test PASSED in 0.4s //tensorflow/core/common_runtime:quantize_training_test PASSED in 2.5s //tensorflow/core/common_runtime:rendezvous_util_test PASSED in 0.1s //tensorflow/core/common_runtime:replicate_per_replica_nodes_test PASSED in 0.9s //tensorflow/core/common_runtime:request_cost_accessor_registry_test PASSED in 1.9s //tensorflow/core/common_runtime:request_cost_test PASSED in 0.1s //tensorflow/core/common_runtime:ring_gatherer_test_cpu PASSED in 3.9s //tensorflow/core/common_runtime:ring_reducer_test_cpu PASSED in 13.1s //tensorflow/core/common_runtime:scoped_allocator_mgr_test PASSED in 18.6s //tensorflow/core/common_runtime:session_test PASSED in 0.8s //tensorflow/core/common_runtime:shape_refiner_test PASSED in 0.7s //tensorflow/core/common_runtime:single_threaded_executor_test PASSED in 0.7s //tensorflow/core/common_runtime:threadpool_device_test PASSED in 1.1s //tensorflow/core/common_runtime:type_inference_test PASSED in 2.1s //tensorflow/core/common_runtime/eager:attr_builder_test PASSED in 28.4s //tensorflow/core/common_runtime/eager:context_test PASSED in 12.6s //tensorflow/core/common_runtime/eager:custom_device_test PASSED in 11.1s //tensorflow/core/common_runtime/eager:eager_executor_test PASSED in 12.2s //tensorflow/core/common_runtime/eager:eager_op_rewrite_registry_test PASSED in 1.8s //tensorflow/core/common_runtime/eager:eager_operation_test PASSED in 11.5s //tensorflow/core/common_runtime/eager:execute_node_test PASSED in 9.7s //tensorflow/core/common_runtime/eager:execute_test PASSED in 28.4s //tensorflow/core/common_runtime/eager:kernel_and_device_test PASSED in 0.8s //tensorflow/core/common_runtime/eager:mkl_eager_op_rewrite_test PASSED in 12.6s //tensorflow/core/common_runtime/eager:placement_test PASSED in 12.2s //tensorflow/core/common_runtime/eager:placement_utils_test PASSED in 13.2s //tensorflow/core/common_runtime/eager:summary_optimizer_test PASSED in 0.3s //tensorflow/core/common_runtime/eager:tensor_handle_data_test PASSED in 15.7s //tensorflow/core/common_runtime/eager:tensor_handle_test PASSED in 12.7s //tensorflow/core/common_runtime/gpu:gpu_device_on_non_gpu_machine_test PASSED in 0.3s //tensorflow/core/common_runtime/next_pluggable_device/c:plugin_c_api_test PASSED in 32.0s //tensorflow/core/config:flags_py_test PASSED in 10.0s //tensorflow/core/config:flags_test PASSED in 0.2s //tensorflow/core/data:compression_utils_test PASSED in 3.0s //tensorflow/core/data:dataset_utils_test PASSED in 0.8s //tensorflow/core/data:hash_utils_test PASSED in 0.7s //tensorflow/core/data:metric_utils_test PASSED in 5.7s //tensorflow/core/data:name_utils_test PASSED in 0.1s //tensorflow/core/data:rewrite_utils_test PASSED in 0.7s //tensorflow/core/data:serialization_utils_test PASSED in 0.6s //tensorflow/core/data:snapshot_utils_test PASSED in 0.7s //tensorflow/core/data:split_utils_test PASSED in 0.5s //tensorflow/core/data:standalone_save_restore_test PASSED in 1.8s //tensorflow/core/data:standalone_test PASSED in 3.6s //tensorflow/core/data:tfdataz_metrics_test PASSED in 1.5s //tensorflow/core/data:unbounded_thread_pool_test PASSED in 0.4s //tensorflow/core/data/service:auto_scaler_test PASSED in 0.1s //tensorflow/core/data/service:auto_shard_rewriter_test PASSED in 0.7s //tensorflow/core/data/service:common_test PASSED in 0.1s //tensorflow/core/data/service:credentials_factory_test PASSED in 0.7s //tensorflow/core/data/service:cross_trainer_cache_test PASSED in 1.3s //tensorflow/core/data/service:data_service_test PASSED in 15.3s //tensorflow/core/data/service:data_transfer_test PASSED in 0.9s //tensorflow/core/data/service:dataset_store_test PASSED in 0.8s //tensorflow/core/data/service:dispatcher_client_test PASSED in 4.9s //tensorflow/core/data/service:dispatcher_state_test PASSED in 0.5s //tensorflow/core/data/service:grpc_dispatcher_impl_test PASSED in 3.8s //tensorflow/core/data/service:grpc_util_test PASSED in 1.3s //tensorflow/core/data/service:grpc_worker_impl_test PASSED in 2.2s //tensorflow/core/data/service:journal_test PASSED in 0.5s //tensorflow/core/data/service:logging_utils_test PASSED in 0.1s //tensorflow/core/data/service:task_runner_test PASSED in 4.0s //tensorflow/core/data/service:test_util_test PASSED in 2.3s //tensorflow/core/data/service:url_test PASSED in 0.1s //tensorflow/core/data/service:utils_test PASSED in 0.5s //tensorflow/core/data/service:validate_utils_test PASSED in 0.2s //tensorflow/core/data/service:worker_client_test PASSED in 4.9s //tensorflow/core/data/service:worker_impl_test PASSED in 3.6s //tensorflow/core/data/service/client:data_service_client_test PASSED in 3.7s //tensorflow/core/data/service/client:utils_test PASSED in 14.7s //tensorflow/core/data/service/client:validate_utils_test PASSED in 1.5s //tensorflow/core/data/service/snapshot:distributed_snapshot_test PASSED in 17.5s //tensorflow/core/data/service/snapshot:file_utils_test PASSED in 0.5s //tensorflow/core/data/service/snapshot:path_utils_test PASSED in 0.1s //tensorflow/core/data/service/snapshot:snapshot_manager_test PASSED in 1.9s //tensorflow/core/data/service/snapshot:snapshot_split_provider_test PASSED in 1.1s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_checkpoint_test PASSED in 3.6s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_test PASSED in 3.8s //tensorflow/core/data/service/snapshot:utils_test PASSED in 0.4s //tensorflow/core/debug:debug_graph_utils_test PASSED in 1.6s //tensorflow/core/distributed_runtime:call_options_test PASSED in 0.3s //tensorflow/core/distributed_runtime:cluster_function_library_runtime_test PASSED in 6.0s //tensorflow/core/distributed_runtime:collective_param_resolver_distributed_test PASSED in 0.9s //tensorflow/core/distributed_runtime:collective_rma_distributed_test PASSED in 1.1s //tensorflow/core/distributed_runtime:device_resolver_distributed_test PASSED in 1.0s //tensorflow/core/distributed_runtime:message_wrappers_test PASSED in 1.4s //tensorflow/core/distributed_runtime:partial_run_mgr_test PASSED in 0.3s //tensorflow/core/distributed_runtime:recent_request_ids_test PASSED in 0.1s //tensorflow/core/distributed_runtime:request_id_test PASSED in 0.1s //tensorflow/core/distributed_runtime:rpc_collective_executor_mgr_test PASSED in 0.4s //tensorflow/core/distributed_runtime:server_lib_test PASSED in 0.3s //tensorflow/core/distributed_runtime:session_mgr_test PASSED in 1.0s //tensorflow/core/distributed_runtime:tensor_coding_test PASSED in 0.2s //tensorflow/core/distributed_runtime/coordination:coordination_service_barrier_proxy_test PASSED in 2.5s //tensorflow/core/distributed_runtime/eager:eager_service_impl_test PASSED in 21.3s //tensorflow/core/distributed_runtime/eager:remote_mgr_test PASSED in 11.9s //tensorflow/core/distributed_runtime/integration_test:c_api_multi_client_test_cpu PASSED in 32.7s //tensorflow/core/distributed_runtime/integration_test:c_api_recoverable_jobs_test_cpu PASSED in 40.4s //tensorflow/core/distributed_runtime/integration_test:c_api_session_coordination_test_cpu PASSED in 28.3s //tensorflow/core/distributed_runtime/rpc:grpc_tensor_coding_test PASSED in 2.4s //tensorflow/core/distributed_runtime/rpc:grpc_worker_cache_test PASSED in 0.5s //tensorflow/core/distributed_runtime/rpc/eager:grpc_eager_client_test PASSED in 0.5s //tensorflow/core/example:example_parser_configuration_test PASSED in 1.0s //tensorflow/core/example:feature_util_test PASSED in 0.1s //tensorflow/core/framework:allocator_test PASSED in 3.5s //tensorflow/core/framework:attr_value_util_test PASSED in 0.7s //tensorflow/core/framework:batch_util_test PASSED in 0.6s //tensorflow/core/framework:bfloat16_test PASSED in 1.1s //tensorflow/core/framework:common_shape_fns_test PASSED in 1.0s //tensorflow/core/framework:dataset_test PASSED in 1.0s //tensorflow/core/framework:device_base_test PASSED in 0.9s //tensorflow/core/framework:disable_jit_test PASSED in 1.8s //tensorflow/core/framework:framework_op_gen_lib_test PASSED in 0.1s //tensorflow/core/framework:framework_op_segment_test PASSED in 0.8s //tensorflow/core/framework:framework_resource_var_test PASSED in 0.2s //tensorflow/core/framework:framework_run_handler_test PASSED in 1.2s //tensorflow/core/framework:framework_run_handler_util_test PASSED in 2.2s //tensorflow/core/framework:full_type_inference_util_test PASSED in 1.0s //tensorflow/core/framework:full_type_util_test PASSED in 0.7s //tensorflow/core/framework:function_test PASSED in 0.9s //tensorflow/core/framework:graph_def_util_test PASSED in 2.0s //tensorflow/core/framework:graph_to_functiondef_test PASSED in 0.9s //tensorflow/core/framework:kernel_def_builder_test PASSED in 0.9s //tensorflow/core/framework:kernel_def_util_test PASSED in 0.9s //tensorflow/core/framework:memory_types_test PASSED in 0.9s //tensorflow/core/framework:model_test PASSED in 1.4s //tensorflow/core/framework:node_def_builder_test PASSED in 0.9s //tensorflow/core/framework:node_def_util_test PASSED in 1.0s //tensorflow/core/framework:node_properties_test PASSED in 0.6s //tensorflow/core/framework:op_compatibility_test PASSED in 0.8s //tensorflow/core/framework:op_def_builder_test PASSED in 0.9s //tensorflow/core/framework:op_def_util_test PASSED in 1.8s //tensorflow/core/framework:op_kernel_test PASSED in 1.1s //tensorflow/core/framework:op_registration_test PASSED in 1.0s //tensorflow/core/framework:partial_tensor_shape_test PASSED in 0.7s //tensorflow/core/framework:rendezvous_test PASSED in 3.0s //tensorflow/core/framework:resource_handle_test PASSED in 0.1s //tensorflow/core/framework:resource_mgr_test PASSED in 2.7s //tensorflow/core/framework:resource_op_kernel_test PASSED in 1.9s //tensorflow/core/framework:shape_inference_test PASSED in 0.9s //tensorflow/core/framework:shape_inference_testutil_test PASSED in 1.1s //tensorflow/core/framework:tensor_shape_test PASSED in 6.6s //tensorflow/core/framework:tensor_slice_test PASSED in 1.0s //tensorflow/core/framework:tensor_test PASSED in 30.9s //tensorflow/core/framework:tensor_testutil_test PASSED in 0.9s //tensorflow/core/framework:tensor_util_test PASSED in 0.8s //tensorflow/core/framework:tracking_allocator_test PASSED in 1.0s //tensorflow/core/framework:types_test PASSED in 0.9s //tensorflow/core/framework:variant_op_registry_test PASSED in 19.1s //tensorflow/core/framework:variant_test PASSED in 12.3s //tensorflow/core/framework/registration:registration_test PASSED in 0.5s //tensorflow/core/function/capture:by_ref_capture_test PASSED in 10.9s //tensorflow/core/function/capture:capture_container_test PASSED in 9.1s //tensorflow/core/function/integration_test:side_inputs_manual_api_test PASSED in 25.9s //tensorflow/core/function/integration_test:side_inputs_test PASSED in 22.3s //tensorflow/core/function/polymorphism:function_cache_test PASSED in 10.9s //tensorflow/core/function/polymorphism:function_type_test PASSED in 13.7s //tensorflow/core/function/polymorphism:type_dispatch_test PASSED in 9.6s //tensorflow/core/function/runtime_client:runtime_client_cc_test PASSED in 45.0s //tensorflow/core/function/trace_type:default_types_test PASSED in 10.1s //tensorflow/core/function/trace_type:serialization_test PASSED in 10.5s //tensorflow/core/function/trace_type:trace_type_test PASSED in 14.4s //tensorflow/core/graph:algorithm_test PASSED in 1.5s //tensorflow/core/graph:collective_order_test PASSED in 0.7s //tensorflow/core/graph:control_flow_test PASSED in 0.9s //tensorflow/core/graph:costmodel_test PASSED in 1.1s //tensorflow/core/graph:edgeset_test PASSED in 0.9s //tensorflow/core/graph:graph_debug_info_builder_test PASSED in 0.7s //tensorflow/core/graph:graph_def_builder_test PASSED in 1.4s //tensorflow/core/graph:graph_partition_test PASSED in 0.7s //tensorflow/core/graph:graph_test PASSED in 0.9s //tensorflow/core/graph:node_builder_test PASSED in 1.4s //tensorflow/core/graph:optimizer_cse_test PASSED in 0.9s //tensorflow/core/graph:subgraph_test PASSED in 0.8s //tensorflow/core/graph:tensor_id_test PASSED in 0.9s //tensorflow/core/graph:validate_test PASSED in 0.9s //tensorflow/core/graph/regularization:simple_delete_test PASSED in 0.2s //tensorflow/core/graph/regularization:util_test PASSED in 0.5s //tensorflow/core/grappler:graph_topology_view_test PASSED in 0.1s //tensorflow/core/grappler:graph_view_test PASSED in 1.2s //tensorflow/core/grappler:grappler_item_builder_test PASSED in 1.1s //tensorflow/core/grappler:grappler_item_test PASSED in 1.7s //tensorflow/core/grappler:mutable_graph_view_test PASSED in 1.8s //tensorflow/core/grappler:utils_test PASSED in 2.3s //tensorflow/core/grappler/clusters:single_machine_test PASSED in 23.9s //tensorflow/core/grappler/clusters:virtual_cluster_test PASSED in 1.5s //tensorflow/core/grappler/costs:analytical_cost_estimator_test PASSED in 1.3s //tensorflow/core/grappler/costs:cost_estimator_test PASSED in 0.1s //tensorflow/core/grappler/costs:graph_memory_test PASSED in 1.6s //tensorflow/core/grappler/costs:graph_properties_test PASSED in 6.2s //tensorflow/core/grappler/costs:robust_stats_test PASSED in 0.1s //tensorflow/core/grappler/costs:utils_test PASSED in 1.6s //tensorflow/core/grappler/costs:virtual_placer_test PASSED in 0.3s //tensorflow/core/grappler/costs:virtual_scheduler_test PASSED in 1.5s //tensorflow/core/grappler/graph_analyzer:gen_node_test PASSED in 2.0s //tensorflow/core/grappler/graph_analyzer:graph_analyzer_test PASSED in 2.8s //tensorflow/core/grappler/graph_analyzer:hash_tools_test PASSED in 1.9s //tensorflow/core/grappler/graph_analyzer:sig_node_test PASSED in 2.5s //tensorflow/core/grappler/graph_analyzer:subgraph_test PASSED in 2.1s //tensorflow/core/grappler/inputs:utils_test PASSED in 0.4s //tensorflow/core/grappler/optimizers:arithmetic_optimizer_test_cpu PASSED in 4.0s //tensorflow/core/grappler/optimizers:auto_mixed_precision_test_cpu PASSED in 1.7s //tensorflow/core/grappler/optimizers:auto_parallel_test_cpu PASSED in 2.5s //tensorflow/core/grappler/optimizers:common_subgraph_elimination_test_cpu PASSED in 1.5s //tensorflow/core/grappler/optimizers:custom_graph_optimizer_registry_test_cpu PASSED in 5.1s //tensorflow/core/grappler/optimizers:debug_stripper_test_cpu PASSED in 2.4s //tensorflow/core/grappler/optimizers:dependency_optimizer_test_cpu PASSED in 1.7s //tensorflow/core/grappler/optimizers:evaluation_utils_test PASSED in 1.4s //tensorflow/core/grappler/optimizers:function_api_info_test PASSED in 0.2s //tensorflow/core/grappler/optimizers:function_optimizer_test_cpu PASSED in 3.0s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_test_cpu PASSED in 1.9s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_factory_test PASSED in 0.2s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_test_cpu PASSED in 2.1s //tensorflow/core/grappler/optimizers:graph_optimizer_stage_test_cpu PASSED in 4.0s //tensorflow/core/grappler/optimizers:implementation_selector_test PASSED in 2.1s //tensorflow/core/grappler/optimizers:loop_optimizer_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:memory_optimizer_test_cpu PASSED in 2.2s //tensorflow/core/grappler/optimizers:meta_optimizer_test_cpu PASSED in 7.6s //tensorflow/core/grappler/optimizers:mkl_remapper_test PASSED in 2.1s //tensorflow/core/grappler/optimizers:model_pruner_test_cpu PASSED in 1.7s //tensorflow/core/grappler/optimizers:pin_to_host_optimizer_test_cpu PASSED in 2.4s //tensorflow/core/grappler/optimizers:remapper_test_cpu PASSED in 2.5s //tensorflow/core/grappler/optimizers:scoped_allocator_optimizer_test PASSED in 2.5s //tensorflow/core/grappler/optimizers:shape_optimizer_test_cpu PASSED in 1.4s //tensorflow/core/grappler/optimizers:static_schedule_test_cpu PASSED in 1.4s //tensorflow/core/grappler/optimizers:tfg_optimizer_hook_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:auto_shard_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:autotune_buffer_sizes_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:batch_parallelization_test PASSED in 0.9s //tensorflow/core/grappler/optimizers/data:disable_intra_op_parallelism_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:disable_prefetch_legacy_autotune_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:enable_gradient_descent_test PASSED in 1.1s //tensorflow/core/grappler/optimizers/data:filter_fusion_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:filter_parallelization_test PASSED in 1.4s //tensorflow/core/grappler/optimizers/data:function_utils_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:fusion_utils_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:graph_utils_test PASSED in 0.9s //tensorflow/core/grappler/optimizers/data:inject_prefetch_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:make_deterministic_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:make_sloppy_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:map_and_batch_fusion_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:map_and_filter_fusion_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:map_fusion_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:map_parallelization_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:noop_elimination_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:parallel_batch_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:replicate_on_split_test PASSED in 1.3s //tensorflow/core/grappler/optimizers/data:shuffle_and_repeat_fusion_test PASSED in 0.3s //tensorflow/core/grappler/optimizers/data:slack_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:split_utils_test PASSED in 1.2s //tensorflow/core/grappler/optimizers/data:use_private_thread_pool_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/inference:batch_op_rewriter_test PASSED in 0.1s //tensorflow/core/grappler/utils:canonicalizer_test PASSED in 1.2s //tensorflow/core/grappler/utils:colocation_test PASSED in 0.8s //tensorflow/core/grappler/utils:frame_test PASSED in 0.4s //tensorflow/core/grappler/utils:functions_test PASSED in 2.0s //tensorflow/core/grappler/utils:graph_view_internal_test PASSED in 0.8s //tensorflow/core/grappler/utils:graph_view_test PASSED in 1.6s //tensorflow/core/grappler/utils:grappler_test_test PASSED in 6.2s //tensorflow/core/grappler/utils:pattern_utils_test PASSED in 0.7s //tensorflow/core/grappler/utils:scc_test PASSED in 1.3s //tensorflow/core/grappler/utils:symbolic_shapes_test PASSED in 0.1s //tensorflow/core/grappler/utils:topological_sort_test PASSED in 6.0s //tensorflow/core/grappler/utils:tpu_test PASSED in 0.1s //tensorflow/core/grappler/utils:transitive_fanin_test PASSED in 0.7s //tensorflow/core/grappler/utils:traversal_test PASSED in 0.6s //tensorflow/core/grappler/verifiers:structure_verifier_test PASSED in 1.1s //tensorflow/core/ir:interfaces_test PASSED in 0.5s //tensorflow/core/ir:ops_test PASSED in 0.2s //tensorflow/core/ir:shape_inference_utils_test PASSED in 0.3s //tensorflow/core/ir:tf_op_registry_test PASSED in 0.3s //tensorflow/core/ir:tf_op_wrapper_test PASSED in 0.7s //tensorflow/core/ir:utility_test PASSED in 0.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:arg_as_control_ret.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:backedge_segment.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:empty.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:error_during_backedge.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_case_with_attr_inference.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_if_with_attr_inference.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_iterator_get_next_attr_inference.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_underscore_output_shapes.pbtxt.test PASSED in 2.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_while_with_attr_inference.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infeed_dequeue.pbtxt.test PASSED in 1.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_arg_handle_type.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_with_output_shapes.pbtxt.test PASSED in 2.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_arg_name.pbtxt.test PASSED in 1.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_backedge_input_size.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_duplicated_node_name.pbtxt.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_index.pbtxt.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_name.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_attr_key.pbtxt.test PASSED in 1.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_key.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_name.pbtxt.test PASSED in 1.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_op_type.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_func_with_empty_name.pbtxt.test PASSED in 2.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_function_import.pbtxt.test PASSED in 1.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_control_result.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_input.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_name.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_result.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_attr_name.pbtxt.test PASSED in 1.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_named_edge_index.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_handle_data.pbtxt.test PASSED in 1.3s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_input.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result_value.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result_value.pbtxt.test PASSED in 1.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_input.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_two_inputs.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_named_edge_index.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_op_name.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_type_list.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:legacy_call.pbtxt.test PASSED in 1.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_shape.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_zero_constant.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:three_nodes_with_attrs.pbtxt.test PASSED in 3.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:version.pbtxt.test PASSED in 1.8s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:empty.mlir.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:fulltype.mlir.test PASSED in 2.0s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:func_with_no_args_or_results.mlir.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:negative_zero_constant.mlir.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:nested_legacy_call.mlir.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:three_nodes_with_attrs.mlir.test PASSED in 1.3s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:version.mlir.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/saved_model:saved_model_roundtrip_test PASSED in 0.3s //tensorflow/core/ir/tests:attributes.mlir.test PASSED in 1.4s //tensorflow/core/ir/tests:canonicalize.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:compatible_types.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:concrete-ops.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:generic_concrete_ops.mlir.test PASSED in 1.0s //tensorflow/core/ir/tests:invalid-concrete-ops.mlir.test PASSED in 1.9s //tensorflow/core/ir/tests:invalid-preserved-attrs.mlir.test PASSED in 0.7s //tensorflow/core/ir/tests:invalid.mlir.test PASSED in 2.5s //tensorflow/core/ir/tests:invalid_types.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:ops.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:region-invalid-ops.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:region-ops-graph.mlir.test PASSED in 1.0s //tensorflow/core/ir/tests:region-ops.mlir.test PASSED in 1.3s //tensorflow/core/ir/tests:types.mlir.test PASSED in 0.6s //tensorflow/core/ir/types:dialect_test PASSED in 0.3s //tensorflow/core/kernels:as_string_op_test PASSED in 0.9s //tensorflow/core/kernels:basic_ops_benchmark_test PASSED in 0.8s //tensorflow/core/kernels:batch_kernels_env_test PASSED in 0.6s //tensorflow/core/kernels:batch_kernels_test PASSED in 35.6s //tensorflow/core/kernels:bias_op_test PASSED in 0.5s //tensorflow/core/kernels:bincount_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:broadcast_to_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:cast_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:checkpoint_callback_manager_test PASSED in 2.1s //tensorflow/core/kernels:clustering_ops_test PASSED in 0.5s //tensorflow/core/kernels:composite_tensor_variant_test PASSED in 0.8s //tensorflow/core/kernels:concat_op_test PASSED in 0.7s //tensorflow/core/kernels:constant_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:control_flow_ops_test PASSED in 6.8s //tensorflow/core/kernels:conv_grad_filter_ops_benchmark_test_cpu PASSED in 0.7s //tensorflow/core/kernels:conv_grad_input_ops_benchmark_test_cpu PASSED in 1.0s //tensorflow/core/kernels:conv_ops_benchmark_test_cpu PASSED in 0.8s //tensorflow/core/kernels:conv_ops_test_cpu PASSED in 4.5s //tensorflow/core/kernels:count_ops_test PASSED in 0.6s //tensorflow/core/kernels:cross_op_test PASSED in 1.5s //tensorflow/core/kernels:cwise_ops_test_cpu PASSED in 0.7s //tensorflow/core/kernels:debug_ops_test PASSED in 1.0s //tensorflow/core/kernels:decode_wav_op_test PASSED in 2.1s //tensorflow/core/kernels:deep_conv2d_test PASSED in 0.5s //tensorflow/core/kernels:dequantize_op_test PASSED in 0.7s //tensorflow/core/kernels:diag_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:dynamic_partition_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:dynamic_stitch_op_test_cpu PASSED in 1.1s //tensorflow/core/kernels:eigen_activations_test PASSED in 0.1s //tensorflow/core/kernels:eigen_attention_test PASSED in 0.6s //tensorflow/core/kernels:eigen_backward_cuboid_convolutions_test PASSED in 1.3s //tensorflow/core/kernels:eigen_backward_spatial_convolutions_test PASSED in 0.6s //tensorflow/core/kernels:eigen_benchmark_cpu_test PASSED in 0.1s //tensorflow/core/kernels:eigen_mkldnn_contraction_kernel_test PASSED in 0.7s //tensorflow/core/kernels:eigen_pooling_test PASSED in 0.6s //tensorflow/core/kernels:encode_wav_op_test PASSED in 3.6s //tensorflow/core/kernels:fingerprint_op_test PASSED in 0.5s //tensorflow/core/kernels:fused_batch_norm_ex_op_test_cpu PASSED in 1.1s //tensorflow/core/kernels:fused_batch_norm_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:gather_nd_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:gather_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:guarantee_const_op_test PASSED in 0.8s //tensorflow/core/kernels:identity_n_op_test PASSED in 0.6s //tensorflow/core/kernels:identity_op_test PASSED in 0.6s //tensorflow/core/kernels:immutable_constant_op_test PASSED in 0.8s //tensorflow/core/kernels:in_topk_op_test PASSED in 0.6s //tensorflow/core/kernels:isotonic_regression_op_test PASSED in 0.5s //tensorflow/core/kernels:logging_ops_test PASSED in 1.5s //tensorflow/core/kernels:lookup_ops_test PASSED in 1.1s //tensorflow/core/kernels:loss_test PASSED in 0.1s //tensorflow/core/kernels:lrn_op_test_cpu PASSED in 1.0s //tensorflow/core/kernels:matmul_op_test_cpu PASSED in 4.1s //tensorflow/core/kernels:merge_v2_checkpoints_op_test PASSED in 1.0s //tensorflow/core/kernels:mfcc_dct_test PASSED in 0.2s //tensorflow/core/kernels:mfcc_mel_filterbank_test PASSED in 0.3s //tensorflow/core/kernels:mfcc_op_test_cpu PASSED in 2.4s //tensorflow/core/kernels:mfcc_test PASSED in 0.2s //tensorflow/core/kernels:multinomial_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:nn_ops_test_cpu PASSED in 0.8s //tensorflow/core/kernels:one_hot_op_test PASSED in 0.5s //tensorflow/core/kernels:ops_testutil_test PASSED in 0.5s //tensorflow/core/kernels:ops_util_test PASSED in 0.1s //tensorflow/core/kernels:parameterized_truncated_normal_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:parse_tensor_test PASSED in 1.2s //tensorflow/core/kernels:quantization_utils_test PASSED in 1.4s //tensorflow/core/kernels:quantize_and_dequantize_op_test_cpu PASSED in 1.4s //tensorflow/core/kernels:quantize_down_and_shrink_range_op_test PASSED in 0.6s //tensorflow/core/kernels:quantize_op_test PASSED in 1.1s //tensorflow/core/kernels:quantized_activation_ops_test PASSED in 0.6s //tensorflow/core/kernels:quantized_add_op_test PASSED in 1.0s //tensorflow/core/kernels:quantized_batch_norm_op_test PASSED in 0.9s //tensorflow/core/kernels:quantized_bias_add_op_test PASSED in 2.4s //tensorflow/core/kernels:quantized_concat_op_test PASSED in 0.7s //tensorflow/core/kernels:quantized_conv_ops_test PASSED in 0.9s //tensorflow/core/kernels:quantized_instance_norm_test PASSED in 1.4s //tensorflow/core/kernels:quantized_matmul_op_test PASSED in 0.8s //tensorflow/core/kernels:quantized_mul_op_test PASSED in 1.3s //tensorflow/core/kernels:quantized_pooling_ops_test PASSED in 0.8s //tensorflow/core/kernels:quantized_reshape_op_test PASSED in 0.6s //tensorflow/core/kernels:quantized_resize_bilinear_op_test PASSED in 2.6s //tensorflow/core/kernels:ragged_fill_empty_rows_op_test PASSED in 0.6s //tensorflow/core/kernels:ragged_gather_op_test PASSED in 0.7s //tensorflow/core/kernels:ragged_range_op_test PASSED in 0.7s //tensorflow/core/kernels:ragged_tensor_from_variant_op_test PASSED in 0.7s //tensorflow/core/kernels:ragged_tensor_to_sparse_kernel_test PASSED in 1.3s //tensorflow/core/kernels:ragged_tensor_to_tensor_op_test PASSED in 0.7s //tensorflow/core/kernels:ragged_tensor_to_variant_op_test PASSED in 1.2s //tensorflow/core/kernels:random_binomial_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:random_index_shuffle_test PASSED in 0.2s //tensorflow/core/kernels:random_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:random_poisson_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:range_sampler_test PASSED in 0.2s //tensorflow/core/kernels:reduction_ops_test_cpu PASSED in 14.5s //tensorflow/core/kernels:regex_replace_op_test PASSED in 0.6s //tensorflow/core/kernels:requantization_range_op_test PASSED in 7.1s //tensorflow/core/kernels:requantize_op_test PASSED in 1.1s //tensorflow/core/kernels:resource_ops_test PASSED in 1.1s //tensorflow/core/kernels:restore_op_test PASSED in 0.9s //tensorflow/core/kernels:restore_v2_op_test PASSED in 0.6s //tensorflow/core/kernels:reverse_op_test PASSED in 0.8s //tensorflow/core/kernels:roll_op_test PASSED in 0.4s //tensorflow/core/kernels:save_op_test PASSED in 0.5s //tensorflow/core/kernels:save_v2_op_test PASSED in 0.6s //tensorflow/core/kernels:scan_ops_test_cpu PASSED in 0.4s //tensorflow/core/kernels:scatter_nd_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:scatter_op_test PASSED in 0.6s //tensorflow/core/kernels:scoped_allocator_ops_test_cpu PASSED in 7.9s //tensorflow/core/kernels:sdca_ops_test PASSED in 1.6s //tensorflow/core/kernels:segment_reduction_ops_test PASSED in 0.4s //tensorflow/core/kernels:sendrecv_ops_test PASSED in 0.8s //tensorflow/core/kernels:sequence_ops_test PASSED in 0.5s //tensorflow/core/kernels:shape_ops_test PASSED in 0.9s //tensorflow/core/kernels:slice_op_test PASSED in 2.1s //tensorflow/core/kernels:spacetobatch_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels:sparse_add_op_test PASSED in 0.6s //tensorflow/core/kernels:sparse_dense_binary_op_shared_test PASSED in 0.7s //tensorflow/core/kernels:sparse_fill_empty_rows_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:sparse_matmul_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:sparse_reduce_sum_op_test PASSED in 0.8s //tensorflow/core/kernels:sparse_tensor_dense_matmul_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:sparse_to_dense_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:sparse_utils_test PASSED in 0.6s //tensorflow/core/kernels:sparse_xent_op_test_cpu PASSED in 3.9s //tensorflow/core/kernels:spectrogram_op_test_cpu PASSED in 2.2s //tensorflow/core/kernels:spectrogram_test PASSED in 0.5s //tensorflow/core/kernels:split_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels:split_v_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:strided_slice_op_test PASSED in 0.6s //tensorflow/core/kernels:string_format_op_test PASSED in 0.6s //tensorflow/core/kernels:string_ngrams_op_test PASSED in 1.1s //tensorflow/core/kernels:string_split_op_test PASSED in 0.5s //tensorflow/core/kernels:substr_op_test PASSED in 0.6s //tensorflow/core/kernels:summary_audio_op_test PASSED in 0.6s //tensorflow/core/kernels:summary_image_op_test PASSED in 0.9s //tensorflow/core/kernels:summary_op_test PASSED in 0.6s //tensorflow/core/kernels:summary_tensor_op_test PASSED in 0.6s //tensorflow/core/kernels:tensor_cord_test PASSED in 0.7s //tensorflow/core/kernels:tensor_flag_utils_test PASSED in 0.1s //tensorflow/core/kernels:tensor_map_test PASSED in 0.2s //tensorflow/core/kernels:training_ops_test PASSED in 0.6s //tensorflow/core/kernels:transpose_util_test PASSED in 1.1s //tensorflow/core/kernels:unary_ops_composition_test_cpu PASSED in 2.0s //tensorflow/core/kernels:unique_op_test PASSED in 0.5s //tensorflow/core/kernels:variable_ops_test PASSED in 1.5s //tensorflow/core/kernels:while_op_test PASSED in 1.1s //tensorflow/core/kernels:xent_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels/batching_util:basic_batch_scheduler_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:batch_input_task_test PASSED in 0.5s //tensorflow/core/kernels/batching_util:batch_resource_base_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:batch_scheduler_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:bounded_executor_test PASSED in 20.3s //tensorflow/core/kernels/batching_util:input_split_metadata_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:periodic_function_test PASSED in 1.6s //tensorflow/core/kernels/batching_util:serial_device_batch_scheduler_test PASSED in 1.6s //tensorflow/core/kernels/batching_util:shared_batch_scheduler_test PASSED in 4.1s //tensorflow/core/kernels/batching_util:threadsafe_status_test PASSED in 0.1s //tensorflow/core/kernels/data:batch_dataset_op_test PASSED in 2.0s //tensorflow/core/kernels/data:cache_dataset_ops_test PASSED in 0.9s //tensorflow/core/kernels/data:concatenate_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data:filter_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:finalize_dataset_op_test PASSED in 2.9s //tensorflow/core/kernels/data:fixed_length_record_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:flat_map_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data:get_options_op_test PASSED in 1.0s //tensorflow/core/kernels/data:interleave_dataset_op_test PASSED in 1.9s //tensorflow/core/kernels/data:iterator_ops_test PASSED in 1.4s //tensorflow/core/kernels/data:map_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:map_defun_op_test PASSED in 0.6s //tensorflow/core/kernels/data:optimize_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:options_dataset_op_test PASSED in 0.5s //tensorflow/core/kernels/data:padded_batch_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:parallel_batch_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data:parallel_filter_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:parallel_interleave_dataset_op_test PASSED in 1.4s //tensorflow/core/kernels/data:parallel_map_dataset_op_test PASSED in 6.4s //tensorflow/core/kernels/data:prefetch_autotuner_test PASSED in 0.2s //tensorflow/core/kernels/data:prefetch_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:range_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:reduce_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:repeat_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data:rewrite_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:shard_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:shuffle_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:skip_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:sparse_tensor_slice_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:take_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data:tensor_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data:tensor_slice_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:text_line_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:tf_record_dataset_op_test PASSED in 5.2s //tensorflow/core/kernels/data:window_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:zip_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data/experimental:assert_next_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data/experimental:assert_prev_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data/experimental:auto_shard_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data/experimental:directed_interleave_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data/experimental:list_dataset_op_test PASSED in 1.7s //tensorflow/core/kernels/data/experimental:map_and_batch_dataset_op_test PASSED in 1.9s //tensorflow/core/kernels/data/experimental:parallel_interleave_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data/experimental:random_dataset_op_test PASSED in 0.5s //tensorflow/core/kernels/data/experimental:sampling_dataset_op_test PASSED in 1.4s //tensorflow/core/kernels/data/experimental:save_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data/experimental:unique_dataset_op_test PASSED in 2.1s //tensorflow/core/kernels/image:adjust_contrast_op_benchmark_test_cpu PASSED in 1.0s //tensorflow/core/kernels/image:adjust_contrast_op_test PASSED in 0.8s //tensorflow/core/kernels/image:colorspace_op_test PASSED in 0.8s //tensorflow/core/kernels/image:crop_and_resize_op_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels/image:crop_and_resize_op_test PASSED in 0.6s //tensorflow/core/kernels/image:encode_jpeg_op_test PASSED in 0.6s //tensorflow/core/kernels/image:mirror_pad_op_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels/image:mirror_pad_op_test PASSED in 0.9s //tensorflow/core/kernels/image:non_max_suppression_op_benchmark_test PASSED in 1.2s //tensorflow/core/kernels/image:non_max_suppression_op_test PASSED in 0.6s //tensorflow/core/kernels/image:resize_area_op_test PASSED in 1.2s //tensorflow/core/kernels/image:resize_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels/image:resize_bicubic_op_test PASSED in 3.4s //tensorflow/core/kernels/image:resize_ops_test_cpu PASSED in 3.1s //tensorflow/core/kernels/image:sampling_kernels_test PASSED in 3.7s //tensorflow/core/kernels/image:scale_and_translate_op_test PASSED in 5.7s //tensorflow/core/kernels/linalg:banded_triangular_solve_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels/linalg:matrix_triangular_solve_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels/mkl:mkl_conv_ops_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_dequantize_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_fused_batch_norm_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_fused_ops_test PASSED in 0.3s //tensorflow/core/kernels/mkl:mkl_matmul_op_benchmark PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_qmatmul_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantize_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantized_concat_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_perchannel_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantized_pooling_ops_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_relu_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_requantize_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_swish_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:onednn_nn_ops_benchmark PASSED in 0.1s //tensorflow/core/kernels/sparse:kernels_test PASSED in 0.9s //tensorflow/core/kernels/uniform_quant_ops:math_utils_test PASSED in 0.1s //tensorflow/core/kernels/uniform_quant_ops:tensor_utils_test PASSED in 0.1s //tensorflow/core/kernels/uniform_quant_ops:uniform_dequantize_op_test PASSED in 1.0s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantize_op_test PASSED in 0.6s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_add_op_test PASSED in 0.8s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_clip_by_value_op_test PASSED in 1.1s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_convolution_ops_test PASSED in 0.5s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_dot_ops_test PASSED in 1.4s //tensorflow/core/kernels/uniform_quant_ops:uniform_requantize_op_test PASSED in 0.6s //tensorflow/core/lib/db:sqlite_test PASSED in 0.3s //tensorflow/core/lib/gif:lib_gif_io_test PASSED in 1.4s //tensorflow/core/lib/jpeg:lib_jpeg_jpeg_mem_unittest PASSED in 0.7s //tensorflow/core/ops:cudnn_rnn_ops_test_cc PASSED in 0.6s //tensorflow/core/ops:ops_array_grad_test PASSED in 1.3s //tensorflow/core/ops:ops_math_grad_test PASSED in 3.9s //tensorflow/core/ops:ops_tests PASSED in 0.9s //tensorflow/core/ops/compat:backwards_compatibility_test PASSED in 0.7s //tensorflow/core/platform:__tensorflow_tsl_platform_profile_utils_cpu_utils_test PASSED in 0.1s //tensorflow/core/platform:enable_tf2_utils_test PASSED in 0.2s //tensorflow/core/platform:env_test PASSED in 3.1s //tensorflow/core/platform:fake_python_env_test PASSED in 0.2s //tensorflow/core/platform:file_system_test PASSED in 0.3s //tensorflow/core/platform:platform_strings_test PASSED in 0.1s //tensorflow/core/platform:ram_file_system_test PASSED in 49.0s //tensorflow/core/platform:resource_loader_test PASSED in 0.2s //tensorflow/core/platform:vmodule_benchmark_test PASSED in 0.4s //tensorflow/core/platform:vmodule_test PASSED in 0.2s //tensorflow/core/profiler/backends/cpu:host_tracer_test PASSED in 0.1s //tensorflow/core/profiler/convert:dcn_analysis_test PASSED in 0.1s //tensorflow/core/profiler/convert:dcn_utils_test PASSED in 0.1s //tensorflow/core/profiler/convert:hlo_proto_to_graph_view_test PASSED in 0.3s //tensorflow/core/profiler/convert:hlo_proto_to_memory_visualization_utils_test PASSED in 0.3s //tensorflow/core/profiler/convert:op_stats_to_pod_stats_test PASSED in 0.1s //tensorflow/core/profiler/convert:op_stats_to_pod_viewer_test PASSED in 0.1s //tensorflow/core/profiler/convert:op_stats_to_tf_stats_test PASSED in 0.2s //tensorflow/core/profiler/convert:repository_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_kernel_stats_db_test PASSED in 0.7s //tensorflow/core/profiler/convert:xplane_to_memory_profile_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_op_metrics_db_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_op_stats_test PASSED in 0.3s //tensorflow/core/profiler/convert:xplane_to_step_events_test PASSED in 0.3s //tensorflow/core/profiler/convert:xplane_to_tf_functions_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_tool_names_test PASSED in 0.1s //tensorflow/core/profiler/convert/trace_viewer:trace_viewer_visibility_test PASSED in 0.1s //tensorflow/core/profiler/internal:tfprof_show_test PASSED in 1.3s //tensorflow/core/profiler/internal:tfprof_stats_test PASSED in 1.1s //tensorflow/core/profiler/internal:tfprof_tensor_test PASSED in 1.0s //tensorflow/core/profiler/internal:tfprof_timeline_test PASSED in 0.7s //tensorflow/core/profiler/internal/advisor:tfprof_advisor_test PASSED in 1.2s //tensorflow/core/profiler/lib:profiler_disabled_test PASSED in 0.1s //tensorflow/core/profiler/utils:derived_timeline_test PASSED in 0.2s //tensorflow/core/profiler/utils:kernel_stats_utils_test PASSED in 0.2s //tensorflow/core/profiler/utils:op_metrics_db_utils_test PASSED in 0.3s //tensorflow/core/profiler/utils:step_intersection_test PASSED in 0.1s //tensorflow/core/runtime_fallback/util:type_util_test PASSED in 0.1s //tensorflow/core/summary:schema_test PASSED in 0.2s //tensorflow/core/summary:summary_db_writer_test PASSED in 0.2s //tensorflow/core/summary:summary_file_writer_test PASSED in 0.4s //tensorflow/core/tfrt/common:pjrt_cpu_client_registration_test PASSED in 5.3s //tensorflow/core/tfrt/common:pjrt_state_test PASSED in 7.4s //tensorflow/core/tfrt/common:pjrt_util_test PASSED in 5.6s //tensorflow/core/tfrt/fallback:cost_recorder_test PASSED in 0.2s //tensorflow/core/tfrt/fallback:fallback_state_test PASSED in 0.4s //tensorflow/core/tfrt/graph_executor:config_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/attribute:attribute_test PASSED in 0.4s //tensorflow/core/tfrt/mlrt/bytecode:bytecode_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/bytecode:executable_test PASSED in 0.2s //tensorflow/core/tfrt/mlrt/bytecode:function_test PASSED in 0.3s //tensorflow/core/tfrt/mlrt/bytecode:kernel_test PASSED in 0.2s //tensorflow/core/tfrt/mlrt/bytecode:span_test PASSED in 0.3s //tensorflow/core/tfrt/mlrt/interpreter:context_test PASSED in 0.2s //tensorflow/core/tfrt/mlrt/interpreter:future_test PASSED in 0.2s //tensorflow/core/tfrt/mlrt/interpreter:interpreter_test PASSED in 0.6s //tensorflow/core/tfrt/mlrt/interpreter:register_span_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/interpreter:value_test PASSED in 0.1s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_concurrent_work_queue_test PASSED in 0.7s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_test PASSED in 1.4s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_util_test PASSED in 0.3s //tensorflow/core/tfrt/runtime:channel_test PASSED in 0.2s //tensorflow/core/tfrt/runtime:tf_threadpool_concurrent_work_queue_test PASSED in 0.8s //tensorflow/core/tfrt/runtime:work_queue_interface_test PASSED in 0.1s //tensorflow/core/tfrt/utils:graph_partition_test PASSED in 2.6s //tensorflow/core/transforms:eval_utils_test PASSED in 1.4s //tensorflow/core/transforms:graph_transform_wrapper_test PASSED in 0.2s //tensorflow/core/util:bcast_test PASSED in 0.9s //tensorflow/core/util:command_line_flags_test PASSED in 1.0s //tensorflow/core/util:debug_data_dumper_test PASSED in 0.7s //tensorflow/core/util:debug_events_writer_test PASSED in 0.2s //tensorflow/core/util:dump_graph_test PASSED in 1.2s //tensorflow/core/util:equal_graph_def_test PASSED in 0.9s //tensorflow/core/util:events_writer_test PASSED in 3.2s //tensorflow/core/util:example_proto_fast_parsing_test PASSED in 1.0s //tensorflow/core/util:example_proto_helper_test PASSED in 0.8s //tensorflow/core/util:exec_on_stall_test PASSED in 2.3s //tensorflow/core/util:fake_clock_env_test PASSED in 3.1s //tensorflow/core/util:incremental_barrier_test PASSED in 0.1s //tensorflow/core/util:matmul_bcast_test PASSED in 0.9s //tensorflow/core/util:memmapped_file_system_test PASSED in 1.1s //tensorflow/core/util:overflow_test PASSED in 0.1s //tensorflow/core/util:presized_cuckoo_map_test PASSED in 1.9s //tensorflow/core/util:ragged_to_dense_util_test PASSED in 0.5s //tensorflow/core/util:reffed_status_callback_test PASSED in 0.8s //tensorflow/core/util:reporter_test PASSED in 0.7s //tensorflow/core/util:saved_tensor_slice_util_test PASSED in 1.2s //tensorflow/core/util:semver_test PASSED in 1.2s //tensorflow/core/util:stat_summarizer_test PASSED in 1.1s //tensorflow/core/util:strided_slice_op_test PASSED in 0.8s //tensorflow/core/util:tensor_format_test PASSED in 1.1s //tensorflow/core/util:tensor_slice_reader_test PASSED in 1.5s //tensorflow/core/util:tensor_slice_set_test PASSED in 0.9s //tensorflow/core/util:tensor_slice_util_test PASSED in 0.9s //tensorflow/core/util:tensor_slice_writer_test PASSED in 1.7s //tensorflow/core/util:work_sharder_test PASSED in 1.5s //tensorflow/core/util/ctc:ctc_beam_search_test PASSED in 0.9s //tensorflow/core/util/proto:descriptor_pool_registry_test PASSED in 1.2s //tensorflow/core/util/proto:proto_utils_test PASSED in 0.6s //tensorflow/core/util/quantization:uniform_quant_ops_params_test PASSED in 0.4s //tensorflow/core/util/sparse:sparse_tensor_test PASSED in 0.2s //tensorflow/core/util/tensor_bundle:tensor_bundle_test PASSED in 33.7s //tensorflow/dtensor/mlir:dtensor_location_test PASSED in 0.2s //tensorflow/dtensor/mlir:group_assignment_test PASSED in 0.2s //tensorflow/dtensor/mlir/tests:annotate_global_shape.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:cluster_function_conversion.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:constant_folding.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:decompose_controlflow.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:designate_resource_handle_mesh.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:device_mesh_cluster_coarsening.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:dtensor_all_gather.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_all_scatter.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_combine_optimization.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_lowering.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_scatter_optimization.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_sum_optimization.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:dtensor_alltoall_lowering.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:dtensor_collective_type_lowering.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_layout_must_execute.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:dtensor_layout_to_xla_sharding_op.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_mixed_precision_reduce.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:dtensor_reduce_scatter_lowering.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_remove_dtensorlayout.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:dtensor_replace_auxiliary_layout_op.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:dtensor_replace_relayout_with_identity.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding_default.mlir.test PASSED in 2.9s //tensorflow/dtensor/mlir/tests:dtensor_xla_spmd_integration.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:elide_identity_before_copy_to_mesh.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:function_renaming.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:handle_cross_cluster_dependencies.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:handle_sparsetensors.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:layout_propagation_v2.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:lower_send_recv.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:merge_clusters.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:mesh_propagation.mlir.test PASSED in 1.6s //tensorflow/dtensor/mlir/tests:multi_device_expansion.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:op_to_device_cluster.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:propagate_default_layout.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:propagate_device_id_to_function.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:restore_and_assign.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:restore_shape_inference.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:set_default_sharding.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:sparse_expansion.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:spmd_batchparallel.mlir.test PASSED in 2.6s //tensorflow/dtensor/mlir/tests:spmd_concat.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:spmd_conv.mlir.test PASSED in 1.6s //tensorflow/dtensor/mlir/tests:spmd_einsum.mlir.test PASSED in 1.8s //tensorflow/dtensor/mlir/tests:spmd_expansion.mlir.test PASSED in 3.3s //tensorflow/dtensor/mlir/tests:spmd_io_ops.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:spmd_iterator.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_matmul.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:spmd_random.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_save_restore.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:spmd_segment_sum.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:spmd_slice.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:spmd_softmax_loss.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_squeeze.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:spmd_var_handle.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:tf_dtensor_ops.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:tpu_add_resource_device_attribute.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:tpu_integration.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:undo_merge_const_across_mesh.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:update_tpu_metadata.mlir.test PASSED in 1.2s //tensorflow/dtensor/python/tests:array_ops_test_cpu PASSED in 36.4s //tensorflow/dtensor/python/tests:collective_combine_all_reduce_test_cpu PASSED in 47.6s //tensorflow/dtensor/python/tests:collective_test_cpu PASSED in 30.9s //tensorflow/dtensor/python/tests:config_test_cpu PASSED in 21.6s //tensorflow/dtensor/python/tests:device_test_cpu PASSED in 55.0s //tensorflow/dtensor/python/tests:layout_test_cpu PASSED in 17.3s //tensorflow/dtensor/python/tests:multi_client_test_cpu PASSED in 17.2s //tensorflow/dtensor/python/tests:numpy_util_test_cpu PASSED in 15.1s //tensorflow/dtensor/tests:dtensor_operation_test PASSED in 29.5s //tensorflow/dtensor/tests:executable_manager_test PASSED in 33.1s //tensorflow/dtensor/tests:layout_to_xla_sharding_test PASSED in 0.3s //tensorflow/dtensor/tests:slice_util_test PASSED in 0.2s //tensorflow/dtensor/tests:spmd_expander_test PASSED in 5.7s //tensorflow/dtensor/tests:tensor_layout_test PASSED in 0.1s //tensorflow/examples/adding_an_op:fact_test PASSED in 25.0s //tensorflow/examples/adding_an_op:zero_out_1_test PASSED in 21.9s //tensorflow/examples/adding_an_op:zero_out_2_test PASSED in 35.5s //tensorflow/examples/adding_an_op:zero_out_3_test PASSED in 25.3s //tensorflow/examples/custom_ops_doc/multiplex_1:multiplex_1_test PASSED in 26.8s //tensorflow/examples/custom_ops_doc/multiplex_2:multiplex_2_test_cpu PASSED in 23.1s //tensorflow/examples/custom_ops_doc/multiplex_3:multiplex_3_test PASSED in 24.8s //tensorflow/examples/custom_ops_doc/multiplex_4:multiplex_4_test PASSED in 42.6s //tensorflow/examples/custom_ops_doc/simple_hash_table:simple_hash_table_test PASSED in 26.1s //tensorflow/examples/custom_ops_doc/sleep:sleep_test PASSED in 26.3s //tensorflow/examples/speech_commands:accuracy_utils_test PASSED in 1.6s //tensorflow/examples/speech_commands:models_test PASSED in 23.8s //tensorflow/examples/speech_commands:recognize_commands_test PASSED in 2.4s //tensorflow/examples/wav_to_spectrogram:wav_to_spectrogram_test PASSED in 2.4s //tensorflow/js:ts_op_gen_test PASSED in 0.4s //tensorflow/python/autograph/converters:asserts_test PASSED in 15.5s //tensorflow/python/autograph/converters:break_statements_test PASSED in 18.5s //tensorflow/python/autograph/converters:call_trees_test PASSED in 16.5s //tensorflow/python/autograph/converters:conditional_expressions_test PASSED in 12.1s //tensorflow/python/autograph/converters:continue_statements_test PASSED in 11.5s //tensorflow/python/autograph/converters:control_flow_test PASSED in 16.0s //tensorflow/python/autograph/converters:directives_test PASSED in 10.9s //tensorflow/python/autograph/converters:functions_test PASSED in 9.3s //tensorflow/python/autograph/converters:lists_test PASSED in 12.2s //tensorflow/python/autograph/converters:logical_expressions_test PASSED in 9.7s //tensorflow/python/autograph/converters:return_statements_test PASSED in 12.2s //tensorflow/python/autograph/converters:slices_test PASSED in 11.0s //tensorflow/python/autograph/converters:variables_test PASSED in 11.7s //tensorflow/python/autograph/core:converter_test PASSED in 9.7s //tensorflow/python/autograph/core:function_wrappers_test PASSED in 10.3s //tensorflow/python/autograph/impl:api_test PASSED in 17.1s //tensorflow/python/autograph/impl:conversion_test PASSED in 12.2s //tensorflow/python/autograph/lang:special_functions_test PASSED in 11.0s //tensorflow/python/autograph/operators:conditional_expressions_test PASSED in 9.9s //tensorflow/python/autograph/operators:control_flow_test PASSED in 22.9s //tensorflow/python/autograph/operators:data_structures_test PASSED in 10.1s //tensorflow/python/autograph/operators:exceptions_test PASSED in 9.1s //tensorflow/python/autograph/operators:logical_test PASSED in 11.3s //tensorflow/python/autograph/operators:py_builtins_test PASSED in 18.0s //tensorflow/python/autograph/operators:slices_test PASSED in 12.6s //tensorflow/python/autograph/operators:variables_test PASSED in 12.7s //tensorflow/python/autograph/pyct:anno_test PASSED in 9.7s //tensorflow/python/autograph/pyct:ast_util_test PASSED in 10.5s //tensorflow/python/autograph/pyct:cache_test PASSED in 9.4s //tensorflow/python/autograph/pyct:cfg_test PASSED in 22.9s //tensorflow/python/autograph/pyct:error_utils_test PASSED in 12.5s //tensorflow/python/autograph/pyct:inspect_utils_test PASSED in 12.5s //tensorflow/python/autograph/pyct:loader_test PASSED in 9.0s //tensorflow/python/autograph/pyct:naming_test PASSED in 10.2s //tensorflow/python/autograph/pyct:origin_info_test PASSED in 9.3s //tensorflow/python/autograph/pyct:parser_test PASSED in 10.6s //tensorflow/python/autograph/pyct:pretty_printer_test PASSED in 17.1s //tensorflow/python/autograph/pyct:qual_names_test PASSED in 9.1s //tensorflow/python/autograph/pyct:templates_test PASSED in 10.2s //tensorflow/python/autograph/pyct:transformer_test PASSED in 9.5s //tensorflow/python/autograph/pyct:transpiler_test PASSED in 12.0s //tensorflow/python/autograph/pyct/static_analysis:activity_test PASSED in 12.5s //tensorflow/python/autograph/pyct/static_analysis:liveness_test PASSED in 10.9s //tensorflow/python/autograph/pyct/static_analysis:reaching_definitions_test PASSED in 25.7s //tensorflow/python/autograph/pyct/static_analysis:reaching_fndefs_test PASSED in 10.2s //tensorflow/python/autograph/pyct/static_analysis:type_inference_test PASSED in 10.9s //tensorflow/python/autograph/tests:assertion_test PASSED in 25.2s //tensorflow/python/autograph/tests:basic_ifexp_test PASSED in 36.4s //tensorflow/python/autograph/tests:call_to_builtin_function_test PASSED in 27.1s //tensorflow/python/autograph/tests:call_to_lambda_function_test PASSED in 30.9s //tensorflow/python/autograph/tests:call_to_named_tuple_test PASSED in 23.6s //tensorflow/python/autograph/tests:call_to_numpy_function_test PASSED in 30.5s //tensorflow/python/autograph/tests:call_to_print_function_test PASSED in 25.8s //tensorflow/python/autograph/tests:call_to_tf_api_test PASSED in 27.9s //tensorflow/python/autograph/tests:call_to_user_function_test PASSED in 18.9s //tensorflow/python/autograph/tests:composite_names_in_control_flow_test PASSED in 48.4s //tensorflow/python/autograph/tests:cond_basic_test PASSED in 40.5s //tensorflow/python/autograph/tests:datasets_test PASSED in 32.1s //tensorflow/python/autograph/tests:early_return_test PASSED in 34.3s //tensorflow/python/autograph/tests:ext_slice_test PASSED in 28.2s //tensorflow/python/autograph/tests:generator_test PASSED in 23.5s //tensorflow/python/autograph/tests:logical_expression_test PASSED in 26.7s //tensorflow/python/autograph/tests:loop_basic_test PASSED in 76.0s //tensorflow/python/autograph/tests:loop_control_flow_illegal_cases_test PASSED in 26.1s //tensorflow/python/autograph/tests:loop_created_variables_test PASSED in 32.7s //tensorflow/python/autograph/tests:loop_scoping_test PASSED in 35.1s //tensorflow/python/autograph/tests:loop_with_function_call_test PASSED in 50.9s //tensorflow/python/autograph/tests:loop_with_variable_type_illegal_cases_test PASSED in 28.6s //tensorflow/python/autograph/tests:loop_with_variable_type_test PASSED in 41.7s //tensorflow/python/autograph/tests:nested_control_flow_test PASSED in 50.7s //tensorflow/python/autograph/tests:type_annotations_test PASSED in 34.8s //tensorflow/python/autograph/utils:context_managers_test PASSED in 10.2s //tensorflow/python/autograph/utils:misc_test PASSED in 18.1s //tensorflow/python/autograph/utils:tensor_list_test PASSED in 15.1s //tensorflow/python/autograph/utils:tensors_test PASSED in 23.0s //tensorflow/python/checkpoint:benchmarks_test PASSED in 10.9s //tensorflow/python/checkpoint:checkpoint_management_test_cpu PASSED in 15.3s //tensorflow/python/checkpoint:checkpoint_metrics_test PASSED in 29.8s //tensorflow/python/checkpoint:checkpoint_test PASSED in 27.9s //tensorflow/python/checkpoint:checkpoint_view_test PASSED in 13.2s //tensorflow/python/checkpoint:checkpoint_with_v1_optimizers_test PASSED in 15.1s //tensorflow/python/checkpoint:functional_saver_test_cpu PASSED in 27.2s //tensorflow/python/checkpoint:restore_test PASSED in 12.3s //tensorflow/python/checkpoint:save_util_v1_test PASSED in 11.0s //tensorflow/python/checkpoint:saveable_compat_test PASSED in 12.5s //tensorflow/python/checkpoint:tensor_callable_test PASSED in 11.3s //tensorflow/python/checkpoint:trackable_view_test PASSED in 17.0s //tensorflow/python/client:device_lib_test_cpu PASSED in 12.2s //tensorflow/python/client:events_writer_test PASSED in 14.3s //tensorflow/python/client:session_benchmark_cpu PASSED in 10.0s //tensorflow/python/client:session_list_devices_test PASSED in 9.9s //tensorflow/python/client:session_partial_run_test PASSED in 26.0s //tensorflow/python/client:timeline_test_cpu PASSED in 10.5s //tensorflow/python/client:virtual_gpu_test_cpu PASSED in 13.7s //tensorflow/python/compat:compat_test PASSED in 12.0s //tensorflow/python/compat:disable_v2_behavior_test PASSED in 10.6s //tensorflow/python/compiler/mlir:mlir_test PASSED in 10.9s //tensorflow/python/compiler/tensorrt:trt_convert_test_cpu PASSED in 21.4s //tensorflow/python/compiler/tensorrt/test:batch_matmul_test_cpu PASSED in 11.8s //tensorflow/python/compiler/tensorrt/test:biasadd_matmul_test_cpu PASSED in 13.5s //tensorflow/python/compiler/tensorrt/test:binary_tensor_weight_broadcast_test_cpu PASSED in 13.0s //tensorflow/python/compiler/tensorrt/test:bool_test_cpu PASSED in 14.2s //tensorflow/python/compiler/tensorrt/test:cast_test_cpu PASSED in 13.3s //tensorflow/python/compiler/tensorrt/test:concatenation_test_cpu PASSED in 11.6s //tensorflow/python/compiler/tensorrt/test:const_broadcast_test_cpu PASSED in 13.1s //tensorflow/python/compiler/tensorrt/test:data_dependent_shape_test_cpu PASSED in 11.9s //tensorflow/python/compiler/tensorrt/test:dynamic_input_shapes_test_cpu PASSED in 12.2s //tensorflow/python/compiler/tensorrt/test:identity_output_test_cpu PASSED in 15.3s //tensorflow/python/compiler/tensorrt/test:int32_test_cpu PASSED in 22.4s //tensorflow/python/compiler/tensorrt/test:lru_cache_test_cpu PASSED in 13.3s //tensorflow/python/compiler/tensorrt/test:multi_connection_neighbor_engine_test_cpu PASSED in 13.2s //tensorflow/python/compiler/tensorrt/test:neighboring_engine_test_cpu PASSED in 12.1s //tensorflow/python/compiler/tensorrt/test:quantization_test_cpu PASSED in 12.1s //tensorflow/python/compiler/tensorrt/test:rank_two_test_cpu PASSED in 11.0s //tensorflow/python/compiler/tensorrt/test:reshape_transpose_test_cpu PASSED in 16.5s //tensorflow/python/compiler/tensorrt/test:topk_test_cpu PASSED in 14.1s //tensorflow/python/compiler/tensorrt/test:trt_engine_op_shape_test_cpu PASSED in 14.0s //tensorflow/python/compiler/tensorrt/test:trt_mode_test_cpu PASSED in 15.2s //tensorflow/python/compiler/tensorrt/test:unary_test_cpu PASSED in 13.4s //tensorflow/python/compiler/tensorrt/test:vgg_block_nchw_test_cpu PASSED in 11.4s //tensorflow/python/compiler/tensorrt/test:vgg_block_test_cpu PASSED in 13.1s //tensorflow/python/compiler/xla:jit_compile_test_cpu PASSED in 25.1s //tensorflow/python/compiler/xla:jit_test_cpu PASSED in 20.2s //tensorflow/python/compiler/xla:xla_test_cpu PASSED in 25.6s //tensorflow/python/compiler/xla/experimental:xla_sharding_test PASSED in 11.5s //tensorflow/python/data/benchmarks:batch_benchmark PASSED in 10.8s //tensorflow/python/data/benchmarks:filter_benchmark PASSED in 9.8s //tensorflow/python/data/benchmarks:from_tensor_slices_benchmark PASSED in 12.5s //tensorflow/python/data/benchmarks:interleave_benchmark PASSED in 11.5s //tensorflow/python/data/benchmarks:list_files_benchmark PASSED in 13.9s //tensorflow/python/data/benchmarks:map_benchmark PASSED in 12.6s //tensorflow/python/data/benchmarks:meta_benchmark PASSED in 16.3s //tensorflow/python/data/benchmarks:prefetch_benchmark PASSED in 9.8s //tensorflow/python/data/benchmarks:range_benchmark PASSED in 11.5s //tensorflow/python/data/experimental/benchmarks:autotune_benchmark PASSED in 22.9s //tensorflow/python/data/experimental/benchmarks:csv_dataset_benchmark PASSED in 17.1s //tensorflow/python/data/experimental/benchmarks:map_and_batch_benchmark PASSED in 11.5s //tensorflow/python/data/experimental/benchmarks:map_defun_benchmark PASSED in 11.4s //tensorflow/python/data/experimental/benchmarks:matching_files_benchmark PASSED in 11.3s //tensorflow/python/data/experimental/benchmarks:optimize_benchmark PASSED in 11.6s //tensorflow/python/data/experimental/benchmarks:parameter_value_benchmark PASSED in 11.1s //tensorflow/python/data/experimental/benchmarks:rejection_resample_benchmark PASSED in 12.6s //tensorflow/python/data/experimental/benchmarks:snapshot_dataset_benchmark PASSED in 11.7s //tensorflow/python/data/experimental/benchmarks:unbatch_benchmark PASSED in 11.2s //tensorflow/python/data/experimental/kernel_tests:assert_cardinality_test PASSED in 42.7s //tensorflow/python/data/experimental/kernel_tests:assert_next_test PASSED in 11.8s //tensorflow/python/data/experimental/kernel_tests:assert_prev_test PASSED in 14.4s //tensorflow/python/data/experimental/kernel_tests:checkpoint_input_pipeline_hook_test PASSED in 27.4s //tensorflow/python/data/experimental/kernel_tests:compression_ops_test PASSED in 14.9s //tensorflow/python/data/experimental/kernel_tests:copy_to_device_test_cpu PASSED in 18.7s //tensorflow/python/data/experimental/kernel_tests:dense_to_sparse_batch_test PASSED in 23.7s //tensorflow/python/data/experimental/kernel_tests:from_list_test PASSED in 35.5s //tensorflow/python/data/experimental/kernel_tests:io_test PASSED in 67.5s //tensorflow/python/data/experimental/kernel_tests:lookup_ops_test PASSED in 12.3s //tensorflow/python/data/experimental/kernel_tests:make_csv_dataset_test PASSED in 34.8s //tensorflow/python/data/experimental/kernel_tests:make_saveable_from_iterator_test PASSED in 15.0s //tensorflow/python/data/experimental/kernel_tests:make_tf_record_dataset_test PASSED in 64.1s //tensorflow/python/data/experimental/kernel_tests:map_defun_op_test PASSED in 15.1s //tensorflow/python/data/experimental/kernel_tests:matching_files_dataset_test PASSED in 35.9s //tensorflow/python/data/experimental/kernel_tests:model_dataset_test PASSED in 12.4s //tensorflow/python/data/experimental/kernel_tests:non_serializable_test PASSED in 11.0s //tensorflow/python/data/experimental/kernel_tests:pad_to_cardinality_test PASSED in 25.7s //tensorflow/python/data/experimental/kernel_tests:prefetch_to_device_test_cpu PASSED in 16.2s //tensorflow/python/data/experimental/kernel_tests:prefetch_with_slack_test PASSED in 27.6s //tensorflow/python/data/experimental/kernel_tests:shuffle_and_repeat_test PASSED in 40.8s //tensorflow/python/data/experimental/kernel_tests:sleep_test PASSED in 10.5s //tensorflow/python/data/experimental/kernel_tests:tf_record_writer_test PASSED in 25.8s //tensorflow/python/data/experimental/kernel_tests:variant_test PASSED in 25.7s //tensorflow/python/data/experimental/kernel_tests:wrap_unwrap_test_cpu PASSED in 14.9s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_fusion_test PASSED in 35.8s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_parallelization_test PASSED in 58.9s //tensorflow/python/data/experimental/kernel_tests/optimization:grappler_test_cpu PASSED in 15.5s //tensorflow/python/data/experimental/kernel_tests/optimization:make_deterministic_test PASSED in 35.2s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_batch_fusion_test PASSED in 11.1s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_filter_fusion_test PASSED in 25.5s //tensorflow/python/data/experimental/kernel_tests/optimization:map_fusion_test PASSED in 28.0s //tensorflow/python/data/experimental/kernel_tests/optimization:map_parallelization_test PASSED in 13.3s //tensorflow/python/data/experimental/kernel_tests/optimization:noop_elimination_test PASSED in 15.7s //tensorflow/python/data/experimental/kernel_tests/service:multi_device_test PASSED in 20.0s //tensorflow/python/data/experimental/service:server_lib_test PASSED in 12.3s //tensorflow/python/data/kernel_tests:as_numpy_iterator_test PASSED in 11.5s //tensorflow/python/data/kernel_tests:bucket_by_sequence_length_test PASSED in 20.0s //tensorflow/python/data/kernel_tests:cache_test PASSED in 49.3s //tensorflow/python/data/kernel_tests:cardinality_test PASSED in 27.9s //tensorflow/python/data/kernel_tests:checkpoint_test PASSED in 25.8s //tensorflow/python/data/kernel_tests:concatenate_test PASSED in 27.9s //tensorflow/python/data/kernel_tests:counter_test PASSED in 28.7s //tensorflow/python/data/kernel_tests:dataset_spec_test PASSED in 15.4s //tensorflow/python/data/kernel_tests:dataset_test PASSED in 31.0s //tensorflow/python/data/kernel_tests:enumerate_test PASSED in 29.6s //tensorflow/python/data/kernel_tests:from_sparse_tensor_slices_test PASSED in 11.3s //tensorflow/python/data/kernel_tests:from_tensor_slices_test PASSED in 48.1s //tensorflow/python/data/kernel_tests:from_tensors_test PASSED in 38.1s //tensorflow/python/data/kernel_tests:get_single_element_test PASSED in 15.2s //tensorflow/python/data/kernel_tests:ignore_errors_test PASSED in 25.9s //tensorflow/python/data/kernel_tests:io_test PASSED in 53.1s //tensorflow/python/data/kernel_tests:iterator_test_cpu PASSED in 17.0s //tensorflow/python/data/kernel_tests:len_test PASSED in 10.1s //tensorflow/python/data/kernel_tests:list_files_test PASSED in 14.4s //tensorflow/python/data/kernel_tests:optional_test_cpu PASSED in 19.6s //tensorflow/python/data/kernel_tests:options_test PASSED in 16.4s //tensorflow/python/data/kernel_tests:placement_test_cpu PASSED in 14.3s //tensorflow/python/data/kernel_tests:prefetch_test PASSED in 73.6s //tensorflow/python/data/kernel_tests:random_test PASSED in 37.2s //tensorflow/python/data/kernel_tests:range_test PASSED in 52.9s //tensorflow/python/data/kernel_tests:rebatch_test PASSED in 11.1s //tensorflow/python/data/kernel_tests:reduce_test_cpu PASSED in 40.3s //tensorflow/python/data/kernel_tests:scan_test_cpu PASSED in 48.6s //tensorflow/python/data/kernel_tests:sparse_batch_test PASSED in 36.9s //tensorflow/python/data/kernel_tests:unbatch_test PASSED in 60.4s //tensorflow/python/data/util:convert_test PASSED in 10.5s //tensorflow/python/data/util:nest_test PASSED in 10.6s //tensorflow/python/data/util:options_test PASSED in 10.5s //tensorflow/python/data/util:random_seed_test PASSED in 11.5s //tensorflow/python/data/util:sparse_test PASSED in 24.2s //tensorflow/python/data/util:structure_test PASSED in 15.5s //tensorflow/python/data/util:traverse_test PASSED in 10.4s //tensorflow/python/debug/cli:analyzer_cli_test_cpu PASSED in 25.3s //tensorflow/python/debug/cli:cli_config_test PASSED in 15.8s //tensorflow/python/debug/cli:cli_shared_test PASSED in 16.0s //tensorflow/python/debug/cli:command_parser_test PASSED in 10.9s //tensorflow/python/debug/cli:debugger_cli_common_test PASSED in 10.1s //tensorflow/python/debug/cli:evaluator_test PASSED in 10.6s //tensorflow/python/debug/cli:profile_analyzer_cli_test PASSED in 10.0s //tensorflow/python/debug/cli:readline_ui_test PASSED in 10.8s //tensorflow/python/debug/cli:tensor_format_test PASSED in 10.1s //tensorflow/python/debug/lib:check_numerics_callback_test_cpu PASSED in 16.1s //tensorflow/python/debug/lib:common_test PASSED in 9.7s //tensorflow/python/debug/lib:debug_data_test PASSED in 10.4s //tensorflow/python/debug/lib:debug_events_monitors_test PASSED in 23.1s //tensorflow/python/debug/lib:debug_events_writer_test PASSED in 10.7s //tensorflow/python/debug/lib:debug_gradients_test_cpu PASSED in 10.4s //tensorflow/python/debug/lib:debug_graph_reconstruction_test_cpu PASSED in 12.2s //tensorflow/python/debug/lib:debug_graphs_test PASSED in 22.9s //tensorflow/python/debug/lib:debug_grappler_test_cpu PASSED in 10.8s //tensorflow/python/debug/lib:debug_utils_test PASSED in 10.8s //tensorflow/python/debug/lib:debug_v2_ops_test_cpu PASSED in 24.9s //tensorflow/python/debug/lib:profiling_test PASSED in 10.7s //tensorflow/python/debug/lib:session_debug_file_test_cpu PASSED in 22.5s //tensorflow/python/debug/lib:session_debug_multi_gpu_test_cpu PASSED in 13.2s //tensorflow/python/debug/lib:source_utils_test PASSED in 23.0s //tensorflow/python/debug/wrappers:disk_usage_test PASSED in 10.0s //tensorflow/python/debug/wrappers:dumping_wrapper_test PASSED in 10.1s //tensorflow/python/debug/wrappers:framework_test PASSED in 14.5s //tensorflow/python/debug/wrappers:local_cli_wrapper_test PASSED in 10.5s //tensorflow/python/distribute:checkpoint_utils_test_2gpu PASSED in 20.9s //tensorflow/python/distribute:checkpoint_utils_test_cpu PASSED in 14.3s //tensorflow/python/distribute:checkpointing_test_2gpu PASSED in 12.3s //tensorflow/python/distribute:checkpointing_test_cpu PASSED in 18.8s //tensorflow/python/distribute:collective_util_test PASSED in 10.1s //tensorflow/python/distribute:combinations_test_2gpu PASSED in 31.5s //tensorflow/python/distribute:combinations_test_cpu PASSED in 26.2s //tensorflow/python/distribute:cross_device_utils_test_cpu PASSED in 16.2s //tensorflow/python/distribute:custom_training_loop_gradient_test_2gpu PASSED in 16.4s //tensorflow/python/distribute:custom_training_loop_gradient_test_cpu PASSED in 14.0s //tensorflow/python/distribute:device_util_test_cpu PASSED in 15.4s //tensorflow/python/distribute:distribute_coordinator_test PASSED in 20.9s //tensorflow/python/distribute:distribute_lib_test PASSED in 15.1s //tensorflow/python/distribute:distribute_utils_test_2gpu PASSED in 11.9s //tensorflow/python/distribute:distribute_utils_test_cpu PASSED in 11.5s //tensorflow/python/distribute:input_ops_test_cpu PASSED in 16.6s //tensorflow/python/distribute:metrics_v1_test_2gpu PASSED in 37.8s //tensorflow/python/distribute:metrics_v1_test_cpu PASSED in 43.8s //tensorflow/python/distribute:mirrored_values_test_2gpu PASSED in 14.6s //tensorflow/python/distribute:mirrored_values_test_cpu PASSED in 14.6s //tensorflow/python/distribute:mirrored_variable_test_2gpu PASSED in 30.4s //tensorflow/python/distribute:mirrored_variable_test_cpu PASSED in 24.7s //tensorflow/python/distribute:multi_process_runner_no_init_test PASSED in 11.1s //tensorflow/python/distribute:multi_worker_continuous_run_test_cpu PASSED in 23.6s //tensorflow/python/distribute:multi_worker_util_test PASSED in 12.3s //tensorflow/python/distribute:numpy_dataset_test PASSED in 11.3s //tensorflow/python/distribute:one_device_strategy_test_cpu PASSED in 21.9s //tensorflow/python/distribute:packed_distributed_variable_test PASSED in 11.5s //tensorflow/python/distribute:parameter_server_strategy_test_2gpu PASSED in 34.2s //tensorflow/python/distribute:parameter_server_strategy_test_cpu PASSED in 37.2s //tensorflow/python/distribute:parameter_server_strategy_v2_test_2gpu PASSED in 26.0s //tensorflow/python/distribute:parameter_server_strategy_v2_test_cpu PASSED in 27.1s //tensorflow/python/distribute:per_replica_test_2gpu PASSED in 12.7s //tensorflow/python/distribute:per_replica_test_cpu PASSED in 18.2s //tensorflow/python/distribute:ps_values_test_2gpu PASSED in 25.1s //tensorflow/python/distribute:ps_values_test_cpu PASSED in 11.6s //tensorflow/python/distribute:remote_mirrored_strategy_eager_test_cpu PASSED in 12.1s //tensorflow/python/distribute:sharded_variable_test PASSED in 30.9s //tensorflow/python/distribute:shared_variable_creator_test PASSED in 10.9s //tensorflow/python/distribute:strategy_combinations_test_cpu PASSED in 50.9s //tensorflow/python/distribute:template_mirrored_strategy_test_cpu PASSED in 12.7s //tensorflow/python/distribute:test_util_test_2gpu PASSED in 16.4s //tensorflow/python/distribute:test_util_test_cpu PASSED in 22.5s //tensorflow/python/distribute:tf_function_test_2gpu PASSED in 13.6s //tensorflow/python/distribute:tf_function_test_cpu PASSED in 14.1s //tensorflow/python/distribute:values_v2_test_cpu PASSED in 16.5s //tensorflow/python/distribute:warm_starting_util_test_2gpu PASSED in 17.4s //tensorflow/python/distribute:warm_starting_util_test_cpu PASSED in 17.6s //tensorflow/python/distribute/cluster_resolver:base_cluster_resolver_py_test PASSED in 11.1s //tensorflow/python/distribute/cluster_resolver:gce_cluster_resolver_py_test PASSED in 22.6s //tensorflow/python/distribute/cluster_resolver:kubernetes_cluster_resolver_py_test PASSED in 10.8s //tensorflow/python/distribute/cluster_resolver:sagemaker_cluster_resolver_py_test PASSED in 21.3s //tensorflow/python/distribute/cluster_resolver:slurm_cluster_resolver_py_test PASSED in 10.4s //tensorflow/python/distribute/cluster_resolver:tfconfig_cluster_resolver_py_test PASSED in 10.9s //tensorflow/python/distribute/cluster_resolver/tpu:tpu_cluster_resolver_py_test PASSED in 10.5s //tensorflow/python/distribute/coordinator:watchdog_test PASSED in 63.8s //tensorflow/python/distribute/experimental:dtensor_util_test_cpu PASSED in 14.1s //tensorflow/python/distribute/experimental:mirrored_strategy_test_cpu PASSED in 36.3s //tensorflow/python/distribute/experimental:multi_worker_mirrored_strategy_test_cpu PASSED in 49.1s //tensorflow/python/distribute/integration_test:saved_model_test_cpu PASSED in 46.0s //tensorflow/python/distribute/parallel_device:parallel_device_test_cpu PASSED in 15.3s //tensorflow/python/distribute/v1:all_reduce_test PASSED in 60.8s //tensorflow/python/distribute/v1:cross_device_ops_test_2gpu PASSED in 49.4s //tensorflow/python/distribute/v1:cross_device_ops_test_cpu PASSED in 69.1s //tensorflow/python/dlpack:dlpack_test_cpu PASSED in 11.1s //tensorflow/python/eager:backprop_test_cpu PASSED in 136.2s //tensorflow/python/eager:benchmarks_test_cpu PASSED in 10.5s //tensorflow/python/eager:cancellation_test_cpu PASSED in 10.0s //tensorflow/python/eager:context_test_cpu PASSED in 13.0s //tensorflow/python/eager:core_test_cpu PASSED in 23.5s //tensorflow/python/eager:gradient_input_output_exclusions_test PASSED in 56.1s //tensorflow/python/eager:graph_only_ops_test_cpu PASSED in 10.7s //tensorflow/python/eager:lift_to_graph_test PASSED in 12.5s //tensorflow/python/eager:monitoring_test_cpu PASSED in 11.8s //tensorflow/python/eager:ops_test_cpu PASSED in 17.4s //tensorflow/python/eager:profiler_client_test PASSED in 8.8s //tensorflow/python/eager:profiler_test_cpu PASSED in 9.6s //tensorflow/python/eager:pywrap_tfe_test PASSED in 23.7s //tensorflow/python/eager:record_test PASSED in 12.4s //tensorflow/python/eager:remote_benchmarks_test_cpu PASSED in 11.8s //tensorflow/python/eager:run_eager_op_as_function_test_cpu PASSED in 11.1s //tensorflow/python/eager:run_eager_op_as_function_xla_test_cpu PASSED in 10.3s //tensorflow/python/eager:small_constants_optimizer_test_cpu PASSED in 234.5s //tensorflow/python/eager:tensor_test_cpu PASSED in 16.4s //tensorflow/python/eager:wrap_function_device_test_cpu PASSED in 16.0s //tensorflow/python/eager:wrap_function_test PASSED in 14.0s //tensorflow/python/eager/benchmarks:kpi_benchmark_test_cpu PASSED in 26.0s //tensorflow/python/eager/memory_tests:remote_memory_test_cpu PASSED in 18.6s //tensorflow/python/eager/polymorphic_function:argument_naming_test_cpu PASSED in 12.5s //tensorflow/python/eager/polymorphic_function:atomic_function_test_cpu PASSED in 11.4s //tensorflow/python/eager/polymorphic_function:collection_test_cpu PASSED in 10.5s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu PASSED in 11.2s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu_mlir_bridge_test PASSED in 19.1s //tensorflow/python/eager/polymorphic_function:concrete_function_test_cpu PASSED in 13.2s //tensorflow/python/eager/polymorphic_function:function_spec_test PASSED in 10.6s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_jit_test_cpu PASSED in 30.4s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_jit_test_cpu_mlir_bridge_test PASSED in 31.8s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_test_cpu PASSED in 12.7s //tensorflow/python/eager/polymorphic_function:tracing_compilation_test PASSED in 18.9s //tensorflow/python/feature_column:sequence_feature_column_integration_test PASSED in 21.4s //tensorflow/python/feature_column:serialization_test PASSED in 29.4s //tensorflow/python/framework:auto_control_deps_test PASSED in 44.3s //tensorflow/python/framework:c_api_util_test PASSED in 11.4s //tensorflow/python/framework:common_shapes_test PASSED in 8.8s //tensorflow/python/framework:composite_tensor_test PASSED in 12.8s //tensorflow/python/framework:config_test_2gpu PASSED in 15.5s //tensorflow/python/framework:config_test_cpu PASSED in 15.0s //tensorflow/python/framework:constant_op_test PASSED in 12.6s //tensorflow/python/framework:device_spec_test PASSED in 25.7s //tensorflow/python/framework:device_test PASSED in 10.8s //tensorflow/python/framework:dtypes_test PASSED in 25.6s //tensorflow/python/framework:error_interpolation_test PASSED in 11.6s //tensorflow/python/framework:errors_test PASSED in 10.6s //tensorflow/python/framework:extension_type_field_test PASSED in 16.3s //tensorflow/python/framework:extension_type_test PASSED in 16.6s //tensorflow/python/framework:file_system_test PASSED in 10.9s //tensorflow/python/framework:flexible_dtypes_test PASSED in 112.0s //tensorflow/python/framework:function_def_to_graph_test PASSED in 12.7s //tensorflow/python/framework:graph_building_benchmark_cpu PASSED in 10.3s //tensorflow/python/framework:graph_util_test PASSED in 20.4s //tensorflow/python/framework:immutable_dict_test PASSED in 11.4s //tensorflow/python/framework:importer_test PASSED in 12.2s //tensorflow/python/framework:indexed_slices_test PASSED in 12.2s //tensorflow/python/framework:kernels_test PASSED in 21.9s //tensorflow/python/framework:meta_graph_test PASSED in 14.4s //tensorflow/python/framework:node_file_writer_test_cpu PASSED in 21.9s //tensorflow/python/framework:offset_counter_helper_test PASSED in 0.1s //tensorflow/python/framework:op_allowlist_namespace_test PASSED in 6.9s //tensorflow/python/framework:op_callbacks_test_cpu PASSED in 13.8s //tensorflow/python/framework:op_def_library_test PASSED in 15.7s //tensorflow/python/framework:op_def_util_test PASSED in 11.6s //tensorflow/python/framework:ops_enable_eager_test PASSED in 2.4s //tensorflow/python/framework:ops_test PASSED in 23.5s //tensorflow/python/framework:proto_test PASSED in 12.8s //tensorflow/python/framework:py_context_manager_test PASSED in 32.9s //tensorflow/python/framework:python_api_dispatcher_test PASSED in 12.4s //tensorflow/python/framework:python_api_info_test PASSED in 10.6s //tensorflow/python/framework:python_api_parameter_converter_test PASSED in 34.0s //tensorflow/python/framework:python_op_gen_annotation_test PASSED in 6.0s //tensorflow/python/framework:python_op_gen_annotator_test PASSED in 0.2s //tensorflow/python/framework:python_op_gen_test PASSED in 0.1s //tensorflow/python/framework:python_tensor_converter_test PASSED in 10.9s //tensorflow/python/framework:random_seed_test PASSED in 10.2s //tensorflow/python/framework:registry_test PASSED in 12.7s //tensorflow/python/framework:smart_cond_test PASSED in 10.1s //tensorflow/python/framework:sparse_tensor_test PASSED in 11.0s //tensorflow/python/framework:subscribe_test PASSED in 10.0s //tensorflow/python/framework:tensor_shape_test PASSED in 10.5s //tensorflow/python/framework:tensor_test PASSED in 13.1s //tensorflow/python/framework:tensor_util_test PASSED in 12.3s //tensorflow/python/framework:test_combinations_test PASSED in 10.2s //tensorflow/python/framework:test_util_test_cpu PASSED in 18.5s //tensorflow/python/framework:tf2_test PASSED in 10.7s //tensorflow/python/framework:traceable_stack_test PASSED in 12.7s //tensorflow/python/framework:type_spec_test PASSED in 10.3s //tensorflow/python/framework:versions_test PASSED in 12.6s //tensorflow/python/framework:weak_tensor_test PASSED in 12.7s //tensorflow/python/framework/experimental:graph_building_test_cpu PASSED in 10.4s //tensorflow/python/framework/experimental:unified_api_test_cpu PASSED in 15.0s //tensorflow/python/grappler:arithmetic_optimizer_test_cpu PASSED in 10.2s //tensorflow/python/grappler:auto_mixed_precision_test_cpu PASSED in 14.9s //tensorflow/python/grappler:constant_folding_test_cpu PASSED in 20.4s //tensorflow/python/grappler:cost_analyzer_test PASSED in 18.7s //tensorflow/python/grappler:datasets_test PASSED in 12.7s //tensorflow/python/grappler:item_test PASSED in 10.3s //tensorflow/python/grappler:memory_optimizer_test PASSED in 19.7s //tensorflow/python/grappler:model_analyzer_test PASSED in 31.4s //tensorflow/python/grappler:remapper_test_cpu PASSED in 10.2s //tensorflow/python/grappler:tf_optimizer_test PASSED in 27.7s //tensorflow/python/kernel_tests:benchmark_test_cpu PASSED in 13.5s //tensorflow/python/kernel_tests:check_ops_test_cpu PASSED in 35.5s //tensorflow/python/kernel_tests:collective_ops_multi_worker_test PASSED in 33.4s //tensorflow/python/kernel_tests:composite_tensor_ops_test PASSED in 10.2s //tensorflow/python/kernel_tests:critical_section_test_cpu PASSED in 53.8s //tensorflow/python/kernel_tests:garbage_collection_test PASSED in 11.2s //tensorflow/python/kernel_tests:gradient_correctness_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests:histogram_ops_test_cpu PASSED in 11.1s //tensorflow/python/kernel_tests:logging_ops_test_cpu PASSED in 13.4s //tensorflow/python/kernel_tests:numerics_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests:template_test PASSED in 35.7s //tensorflow/python/kernel_tests:trace_op_test_cpu PASSED in 12.4s //tensorflow/python/kernel_tests/array_ops:batch_gather_op_test_cpu PASSED in 23.7s //tensorflow/python/kernel_tests/array_ops:batch_scatter_ops_test PASSED in 11.3s //tensorflow/python/kernel_tests/array_ops:batchtospace_op_test_cpu PASSED in 14.9s //tensorflow/python/kernel_tests/array_ops:bcast_ops_test PASSED in 12.0s //tensorflow/python/kernel_tests/array_ops:bitcast_op_test_cpu PASSED in 11.8s //tensorflow/python/kernel_tests/array_ops:broadcast_to_ops_test_cpu PASSED in 53.9s //tensorflow/python/kernel_tests/array_ops:cast_op_test_cpu PASSED in 16.6s //tensorflow/python/kernel_tests/array_ops:constant_op_eager_test_cpu PASSED in 27.7s //tensorflow/python/kernel_tests/array_ops:constant_op_test_cpu PASSED in 16.8s //tensorflow/python/kernel_tests/array_ops:denormal_test_cpu PASSED in 21.5s //tensorflow/python/kernel_tests/array_ops:depthtospace_op_test_cpu PASSED in 17.3s //tensorflow/python/kernel_tests/array_ops:edit_distance_op_test PASSED in 10.6s //tensorflow/python/kernel_tests/array_ops:fingerprint_op_test PASSED in 11.5s //tensorflow/python/kernel_tests/array_ops:gather_nd_op_test_cpu PASSED in 27.9s //tensorflow/python/kernel_tests/array_ops:identity_n_op_py_test PASSED in 10.1s //tensorflow/python/kernel_tests/array_ops:identity_op_py_test PASSED in 11.6s //tensorflow/python/kernel_tests/array_ops:large_concat_op_test_cpu PASSED in 13.6s //tensorflow/python/kernel_tests/array_ops:manip_ops_test_cpu PASSED in 14.8s //tensorflow/python/kernel_tests/array_ops:one_hot_op_test_cpu PASSED in 11.7s //tensorflow/python/kernel_tests/array_ops:pad_op_test_cpu PASSED in 18.6s //tensorflow/python/kernel_tests/array_ops:reshape_op_test_cpu PASSED in 10.6s //tensorflow/python/kernel_tests/array_ops:reverse_sequence_op_test_cpu PASSED in 14.3s //tensorflow/python/kernel_tests/array_ops:scalar_test_cpu PASSED in 11.7s //tensorflow/python/kernel_tests/array_ops:shape_ops_test_cpu PASSED in 18.4s //tensorflow/python/kernel_tests/array_ops:slice_op_test_cpu PASSED in 17.2s //tensorflow/python/kernel_tests/array_ops:spacetobatch_op_test_cpu PASSED in 18.3s //tensorflow/python/kernel_tests/array_ops:spacetodepth_op_test_cpu PASSED in 12.5s //tensorflow/python/kernel_tests/array_ops:stack_op_test_cpu PASSED in 17.2s //tensorflow/python/kernel_tests/array_ops:unique_op_test_cpu PASSED in 13.8s //tensorflow/python/kernel_tests/array_ops:unstack_op_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/array_ops:where_op_test_cpu PASSED in 20.0s //tensorflow/python/kernel_tests/control_flow:cond_v2_test_cpu PASSED in 62.2s //tensorflow/python/kernel_tests/control_flow:control_flow_util_test PASSED in 13.4s //tensorflow/python/kernel_tests/control_flow:control_flow_util_v2_test PASSED in 31.2s //tensorflow/python/kernel_tests/control_flow:py_func_test_cpu PASSED in 25.6s //tensorflow/python/kernel_tests/control_flow:scan_ops_test_cpu PASSED in 68.9s //tensorflow/python/kernel_tests/control_flow:while_v2_test_cpu PASSED in 79.2s //tensorflow/python/kernel_tests/custom_ops:ackermann_test PASSED in 14.0s //tensorflow/python/kernel_tests/custom_ops:duplicate_op_test PASSED in 17.6s //tensorflow/python/kernel_tests/custom_ops:invalid_op_test PASSED in 10.9s //tensorflow/python/kernel_tests/data_structures:conditional_accumulator_test PASSED in 25.6s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_2gpu PASSED in 37.5s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_cpu PASSED in 14.9s //tensorflow/python/kernel_tests/data_structures:dynamic_stitch_op_test_cpu PASSED in 11.8s //tensorflow/python/kernel_tests/data_structures:fifo_queue_test PASSED in 13.2s //tensorflow/python/kernel_tests/data_structures:list_ops_test_cpu PASSED in 31.6s //tensorflow/python/kernel_tests/data_structures:listdiff_op_test PASSED in 11.4s //tensorflow/python/kernel_tests/data_structures:lookup_ops_test PASSED in 33.8s //tensorflow/python/kernel_tests/data_structures:map_ops_test PASSED in 23.0s //tensorflow/python/kernel_tests/data_structures:padding_fifo_queue_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests/data_structures:priority_queue_test PASSED in 20.9s //tensorflow/python/kernel_tests/data_structures:stack_ops_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/data_structures:stage_op_test_cpu PASSED in 12.6s //tensorflow/python/kernel_tests/distributions:bernoulli_test_cpu PASSED in 18.8s //tensorflow/python/kernel_tests/distributions:bijector_test_cpu PASSED in 11.1s //tensorflow/python/kernel_tests/distributions:categorical_test_cpu PASSED in 16.7s //tensorflow/python/kernel_tests/distributions:dirichlet_multinomial_test_cpu PASSED in 15.3s //tensorflow/python/kernel_tests/distributions:dirichlet_test_cpu PASSED in 18.3s //tensorflow/python/kernel_tests/distributions:exponential_test_cpu PASSED in 17.1s //tensorflow/python/kernel_tests/distributions:gamma_test_cpu PASSED in 92.8s //tensorflow/python/kernel_tests/distributions:identity_bijector_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/distributions:kullback_leibler_test_cpu PASSED in 16.3s //tensorflow/python/kernel_tests/distributions:laplace_test_cpu PASSED in 42.3s //tensorflow/python/kernel_tests/distributions:multinomial_test_cpu PASSED in 10.7s //tensorflow/python/kernel_tests/distributions:normal_test_cpu PASSED in 50.5s //tensorflow/python/kernel_tests/distributions:special_math_test_cpu PASSED in 28.5s //tensorflow/python/kernel_tests/distributions:uniform_test_cpu PASSED in 15.8s //tensorflow/python/kernel_tests/image_ops:attention_ops_test PASSED in 11.2s //tensorflow/python/kernel_tests/image_ops:decode_bmp_op_test PASSED in 17.9s //tensorflow/python/kernel_tests/image_ops:decode_compressed_op_test PASSED in 13.6s //tensorflow/python/kernel_tests/image_ops:decode_image_op_test PASSED in 13.6s //tensorflow/python/kernel_tests/image_ops:decode_jpeg_op_test PASSED in 17.1s //tensorflow/python/kernel_tests/image_ops:decode_png_op_test PASSED in 9.9s //tensorflow/python/kernel_tests/image_ops:decode_raw_op_test PASSED in 11.4s //tensorflow/python/kernel_tests/image_ops:draw_bounding_box_op_test_cpu PASSED in 11.6s //tensorflow/python/kernel_tests/image_ops:extract_image_patches_op_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests/image_ops:extract_volume_patches_op_test_cpu PASSED in 10.6s //tensorflow/python/kernel_tests/io_ops:checkpoint_ops_test PASSED in 12.0s //tensorflow/python/kernel_tests/io_ops:decode_csv_op_test PASSED in 11.1s //tensorflow/python/kernel_tests/io_ops:io_ops_test PASSED in 12.4s //tensorflow/python/kernel_tests/io_ops:parse_single_example_op_test PASSED in 12.8s //tensorflow/python/kernel_tests/io_ops:parsing_ops_test PASSED in 31.9s //tensorflow/python/kernel_tests/io_ops:reader_ops_test PASSED in 15.5s //tensorflow/python/kernel_tests/io_ops:record_input_test PASSED in 28.9s //tensorflow/python/kernel_tests/io_ops:save_restore_ops_test PASSED in 11.1s //tensorflow/python/kernel_tests/linalg:determinant_op_test_cpu PASSED in 22.6s //tensorflow/python/kernel_tests/linalg:linear_operator_addition_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/linalg:linear_operator_algebra_test_cpu PASSED in 11.1s //tensorflow/python/kernel_tests/linalg:linear_operator_test_cpu PASSED in 19.9s //tensorflow/python/kernel_tests/linalg:lu_op_test_cpu PASSED in 14.9s //tensorflow/python/kernel_tests/linalg:matrix_inverse_op_test_cpu PASSED in 14.8s //tensorflow/python/kernel_tests/linalg:matrix_logarithm_op_test PASSED in 77.2s //tensorflow/python/kernel_tests/linalg:matrix_solve_ls_op_test_cpu PASSED in 30.9s //tensorflow/python/kernel_tests/linalg:matrix_solve_op_test_cpu PASSED in 36.6s //tensorflow/python/kernel_tests/linalg:matrix_square_root_op_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests/linalg:slicing_test_cpu PASSED in 18.3s //tensorflow/python/kernel_tests/linalg/sparse:conjugate_gradient_test_cpu PASSED in 14.8s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_test_cpu PASSED in 11.0s //tensorflow/python/kernel_tests/math_ops:aggregate_ops_test_cpu PASSED in 13.0s //tensorflow/python/kernel_tests/math_ops:argmax_op_test_cpu PASSED in 12.8s //tensorflow/python/kernel_tests/math_ops:banded_triangular_solve_op_test_cpu PASSED in 16.3s //tensorflow/python/kernel_tests/math_ops:basic_gpu_test_cpu PASSED in 11.3s //tensorflow/python/kernel_tests/math_ops:bincount_op_test_cpu PASSED in 13.1s //tensorflow/python/kernel_tests/math_ops:bucketize_op_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/math_ops:clip_ops_test PASSED in 13.6s //tensorflow/python/kernel_tests/math_ops:confusion_matrix_test PASSED in 14.3s //tensorflow/python/kernel_tests/math_ops:cross_grad_test_cpu PASSED in 11.6s //tensorflow/python/kernel_tests/math_ops:cumulative_logsumexp_test_cpu PASSED in 11.8s //tensorflow/python/kernel_tests/math_ops:in_topk_op_test_cpu PASSED in 12.8s //tensorflow/python/kernel_tests/math_ops:reduce_benchmark_test_cpu PASSED in 20.9s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_d9m_test_cpu PASSED in 10.5s //tensorflow/python/kernel_tests/math_ops:sets_test PASSED in 28.5s //tensorflow/python/kernel_tests/math_ops:topk_op_test_cpu PASSED in 12.6s //tensorflow/python/kernel_tests/math_ops:zero_division_test_cpu PASSED in 9.8s //tensorflow/python/kernel_tests/nn_ops:betainc_op_test_cpu PASSED in 13.3s //tensorflow/python/kernel_tests/nn_ops:bias_op_test_cpu PASSED in 137.5s //tensorflow/python/kernel_tests/nn_ops:conv1d_test_cpu PASSED in 13.3s //tensorflow/python/kernel_tests/nn_ops:conv1d_transpose_test_cpu PASSED in 10.7s //tensorflow/python/kernel_tests/nn_ops:conv2d_transpose_test_cpu PASSED in 12.5s //tensorflow/python/kernel_tests/nn_ops:conv3d_backprop_filter_v2_grad_test_cpu PASSED in 15.5s //tensorflow/python/kernel_tests/nn_ops:conv3d_transpose_test_cpu PASSED in 12.4s //tensorflow/python/kernel_tests/nn_ops:ctc_decoder_ops_test PASSED in 10.3s //tensorflow/python/kernel_tests/nn_ops:ctc_loss_op_test_cpu PASSED in 71.7s //tensorflow/python/kernel_tests/nn_ops:cudnn_d9m_test_cpu PASSED in 13.6s //tensorflow/python/kernel_tests/nn_ops:cudnn_deterministic_ops_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/nn_ops:losses_test PASSED in 36.5s //tensorflow/python/kernel_tests/nn_ops:lrn_op_test_cpu PASSED in 21.6s //tensorflow/python/kernel_tests/nn_ops:morphological_ops_test_cpu PASSED in 17.1s //tensorflow/python/kernel_tests/nn_ops:nth_element_op_test_cpu PASSED in 13.6s //tensorflow/python/kernel_tests/nn_ops:pool_test_cpu PASSED in 34.9s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_3d_test_cpu PASSED in 29.7s //tensorflow/python/kernel_tests/nn_ops:relu_op_test_cpu PASSED in 15.5s //tensorflow/python/kernel_tests/nn_ops:softmax_op_test_cpu PASSED in 11.5s //tensorflow/python/kernel_tests/nn_ops:softplus_op_test_cpu PASSED in 12.0s //tensorflow/python/kernel_tests/nn_ops:softsign_op_test_cpu PASSED in 13.2s //tensorflow/python/kernel_tests/nn_ops:xent_op_d9m_test_cpu PASSED in 135.7s //tensorflow/python/kernel_tests/nn_ops:xent_op_test_cpu PASSED in 14.7s //tensorflow/python/kernel_tests/proto:descriptor_source_test PASSED in 10.2s //tensorflow/python/kernel_tests/proto:encode_proto_op_test PASSED in 10.3s //tensorflow/python/kernel_tests/quantization_ops:quantization_ops_test PASSED in 12.3s //tensorflow/python/kernel_tests/random:candidate_sampler_ops_test PASSED in 12.1s //tensorflow/python/kernel_tests/random:multinomial_op_test_cpu PASSED in 26.3s //tensorflow/python/kernel_tests/random:parameterized_truncated_normal_op_test_cpu PASSED in 21.8s //tensorflow/python/kernel_tests/random:random_crop_test_cpu PASSED in 11.8s //tensorflow/python/kernel_tests/random:random_grad_test_cpu PASSED in 13.8s //tensorflow/python/kernel_tests/random:random_ops_test_cpu PASSED in 31.0s //tensorflow/python/kernel_tests/random:random_poisson_test_cpu PASSED in 17.3s //tensorflow/python/kernel_tests/random:random_shuffle_queue_test PASSED in 9.8s //tensorflow/python/kernel_tests/random:stateful_random_ops_test_cpu PASSED in 20.3s //tensorflow/python/kernel_tests/signal:mel_ops_test_cpu PASSED in 17.5s //tensorflow/python/kernel_tests/signal:mfcc_ops_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/signal:reconstruction_ops_test_cpu PASSED in 15.7s //tensorflow/python/kernel_tests/signal:shape_ops_test_cpu PASSED in 23.7s //tensorflow/python/kernel_tests/sparse_ops:sparse_add_op_test PASSED in 11.7s //tensorflow/python/kernel_tests/sparse_ops:sparse_concat_op_test PASSED in 11.1s //tensorflow/python/kernel_tests/sparse_ops:sparse_conditional_accumulator_test PASSED in 11.2s //tensorflow/python/kernel_tests/sparse_ops:sparse_cross_op_test PASSED in 16.9s //tensorflow/python/kernel_tests/sparse_ops:sparse_matmul_op_test_cpu PASSED in 40.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_reorder_op_test PASSED in 11.7s //tensorflow/python/kernel_tests/sparse_ops:sparse_reshape_op_test PASSED in 11.9s //tensorflow/python/kernel_tests/sparse_ops:sparse_serialization_ops_test PASSED in 12.1s //tensorflow/python/kernel_tests/sparse_ops:sparse_slice_op_test PASSED in 14.9s //tensorflow/python/kernel_tests/sparse_ops:sparse_split_op_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_grad_test_cpu PASSED in 18.8s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_d9m_test_cpu PASSED in 38.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_test_cpu PASSED in 36.7s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensors_map_ops_test PASSED in 12.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_to_dense_op_py_test_cpu PASSED in 10.7s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_d9m_test_cpu PASSED in 77.8s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests/sparse_ops:sparsemask_op_test PASSED in 9.6s //tensorflow/python/kernel_tests/strings_ops:as_string_op_test PASSED in 10.6s //tensorflow/python/kernel_tests/strings_ops:base64_ops_test PASSED in 14.6s //tensorflow/python/kernel_tests/strings_ops:reduce_join_op_test_cpu PASSED in 22.7s //tensorflow/python/kernel_tests/strings_ops:regex_full_match_op_test PASSED in 12.5s //tensorflow/python/kernel_tests/strings_ops:regex_replace_op_test PASSED in 11.4s //tensorflow/python/kernel_tests/strings_ops:string_bytes_split_op_test PASSED in 14.2s //tensorflow/python/kernel_tests/strings_ops:string_format_op_test PASSED in 16.6s //tensorflow/python/kernel_tests/strings_ops:string_join_op_test PASSED in 11.2s //tensorflow/python/kernel_tests/strings_ops:string_length_op_test PASSED in 10.9s //tensorflow/python/kernel_tests/strings_ops:string_lower_op_test PASSED in 13.7s //tensorflow/python/kernel_tests/strings_ops:string_split_op_test PASSED in 34.2s //tensorflow/python/kernel_tests/strings_ops:string_strip_op_test PASSED in 11.9s //tensorflow/python/kernel_tests/strings_ops:string_to_hash_bucket_op_test_cpu PASSED in 13.5s //tensorflow/python/kernel_tests/strings_ops:string_to_number_op_test_cpu PASSED in 13.4s //tensorflow/python/kernel_tests/strings_ops:string_upper_op_test PASSED in 9.8s //tensorflow/python/kernel_tests/strings_ops:substr_op_test PASSED in 41.2s //tensorflow/python/kernel_tests/strings_ops:unicode_decode_op_test PASSED in 20.9s //tensorflow/python/kernel_tests/strings_ops:unicode_encode_op_test PASSED in 16.7s //tensorflow/python/kernel_tests/strings_ops:unicode_script_op_test PASSED in 10.3s //tensorflow/python/kernel_tests/strings_ops:unicode_transcode_op_test PASSED in 16.8s //tensorflow/python/kernel_tests/strings_ops:unsorted_segment_join_op_test_cpu PASSED in 15.8s //tensorflow/python/kernel_tests/summary_ops:summary_ops_test_cpu PASSED in 18.9s //tensorflow/python/kernel_tests/summary_ops:summary_v1_audio_op_test_cpu PASSED in 11.5s //tensorflow/python/kernel_tests/summary_ops:summary_v1_image_op_test_cpu PASSED in 17.0s //tensorflow/python/kernel_tests/summary_ops:summary_v1_ops_test PASSED in 9.2s //tensorflow/python/kernel_tests/summary_ops:summary_v1_tensor_op_test PASSED in 11.1s //tensorflow/python/kernel_tests/v1_compat_tests:array_ops_test_cpu PASSED in 15.6s //tensorflow/python/kernel_tests/v1_compat_tests:dense_update_ops_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests/v1_compat_tests:identity_op_py_test PASSED in 11.1s //tensorflow/python/kernel_tests/v1_compat_tests:scatter_nd_ops_test_cpu PASSED in 10.5s //tensorflow/python/kernel_tests/v1_compat_tests:session_ops_test_cpu PASSED in 12.1s //tensorflow/python/kernel_tests/v1_compat_tests:stack_op_test_cpu PASSED in 11.3s //tensorflow/python/kernel_tests/variables:dense_update_ops_no_tsan_test_cpu PASSED in 12.7s //tensorflow/python/kernel_tests/variables:dense_update_ops_test_cpu PASSED in 11.6s //tensorflow/python/kernel_tests/variables:partitioned_variables_test PASSED in 15.8s //tensorflow/python/kernel_tests/variables:resource_variable_ops_test_cpu PASSED in 64.4s //tensorflow/python/kernel_tests/variables:variable_ops_test_cpu PASSED in 25.9s //tensorflow/python/kernel_tests/variables:variable_scope_test PASSED in 46.7s //tensorflow/python/kernel_tests/variables:variables_test PASSED in 12.3s //tensorflow/python/lib/io:file_io_test PASSED in 14.1s //tensorflow/python/lib/io:tf_record_test PASSED in 12.3s //tensorflow/python/module:module_test PASSED in 14.4s //tensorflow/python/ops:array_grad_test_cpu PASSED in 13.9s //tensorflow/python/ops:array_ops_shape_test PASSED in 10.8s //tensorflow/python/ops:array_ops_test PASSED in 12.4s //tensorflow/python/ops:autograph_ops_test PASSED in 10.0s //tensorflow/python/ops:batch_norm_benchmark_cpu PASSED in 13.5s //tensorflow/python/ops:bincount_ops_test_cpu PASSED in 14.4s //tensorflow/python/ops:bitwise_ops_test_cpu PASSED in 20.5s //tensorflow/python/ops:clip_ops_test PASSED in 15.2s //tensorflow/python/ops:clustering_ops_test PASSED in 40.8s //tensorflow/python/ops:collective_ops_benchmark_cpu PASSED in 11.2s //tensorflow/python/ops:collective_ops_gpu_test_2gpu PASSED in 11.9s //tensorflow/python/ops:collective_ops_gpu_test_cpu PASSED in 16.0s //tensorflow/python/ops:collective_ops_test PASSED in 18.5s //tensorflow/python/ops:collective_ops_xla_test PASSED in 11.0s //tensorflow/python/ops:compiled_collective_ops_gpu_test_2gpu PASSED in 10.8s //tensorflow/python/ops:compiled_collective_ops_gpu_test_cpu PASSED in 12.7s //tensorflow/python/ops:concat_benchmark_cpu PASSED in 13.7s //tensorflow/python/ops:control_flow_ops_benchmark_cpu PASSED in 19.8s //tensorflow/python/ops:control_flow_v2_enable_test PASSED in 22.1s //tensorflow/python/ops:control_flow_v2_toggles_test PASSED in 8.4s //tensorflow/python/ops:dequantize_op_test PASSED in 10.7s //tensorflow/python/ops:embedding_ops_test_cpu PASSED in 12.9s //tensorflow/python/ops:factory_ops_test_cpu PASSED in 10.2s //tensorflow/python/ops:functional_ops_test PASSED in 9.3s //tensorflow/python/ops:gradient_checker_v2_test_cpu PASSED in 46.0s //tensorflow/python/ops:gradients_test_cpu PASSED in 20.4s //tensorflow/python/ops:init_ops_test_cpu PASSED in 27.2s //tensorflow/python/ops:init_ops_v2_test_cpu PASSED in 13.6s //tensorflow/python/ops:math_grad_test_cpu PASSED in 36.7s //tensorflow/python/ops:math_ops_linspace_test_cpu PASSED in 14.6s //tensorflow/python/ops:math_ops_test_cpu PASSED in 31.2s //tensorflow/python/ops:matmul_benchmark_cpu PASSED in 9.1s //tensorflow/python/ops:nn_grad_test_cpu PASSED in 12.3s //tensorflow/python/ops:nn_loss_scaling_utilities_test PASSED in 21.0s //tensorflow/python/ops:nn_test_cpu PASSED in 63.3s //tensorflow/python/ops:nn_xent_test_cpu PASSED in 10.5s //tensorflow/python/ops:op_selector_test PASSED in 10.4s //tensorflow/python/ops:quantized_conv_ops_test PASSED in 13.3s //tensorflow/python/ops:quantized_ops_test PASSED in 11.6s //tensorflow/python/ops:raw_ops_test_cpu PASSED in 10.9s //tensorflow/python/ops:rnn_grad_test_cpu PASSED in 26.6s //tensorflow/python/ops:script_ops_test PASSED in 16.5s //tensorflow/python/ops:sort_ops_test PASSED in 26.5s //tensorflow/python/ops:sparse_bincount_ops_test_cpu PASSED in 15.9s //tensorflow/python/ops:sparse_ops_test PASSED in 22.4s //tensorflow/python/ops:split_benchmark_cpu PASSED in 8.6s //tensorflow/python/ops:tensor_array_ops_test PASSED in 10.5s //tensorflow/python/ops:transpose_benchmark_cpu PASSED in 14.0s //tensorflow/python/ops:variable_spec_test PASSED in 24.7s //tensorflow/python/ops:weak_tensor_array_ops_test PASSED in 11.6s //tensorflow/python/ops:weak_tensor_image_ops_test PASSED in 9.7s //tensorflow/python/ops:weak_tensor_math_ops_test PASSED in 13.7s //tensorflow/python/ops:weak_tensor_nn_test_cpu PASSED in 33.5s //tensorflow/python/ops:weak_tensor_np_array_ops_test PASSED in 32.0s //tensorflow/python/ops:weak_tensor_np_math_ops_test PASSED in 11.7s //tensorflow/python/ops:weak_tensor_ops_test PASSED in 19.9s //tensorflow/python/ops/losses:util_test PASSED in 9.2s //tensorflow/python/ops/memory_tests:custom_gradient_memory_test_cpu PASSED in 13.6s //tensorflow/python/ops/numpy_ops:np_array_ops_test_cpu PASSED in 77.1s //tensorflow/python/ops/numpy_ops:np_arrays_test_cpu PASSED in 11.0s //tensorflow/python/ops/numpy_ops:np_dtypes_test_cpu PASSED in 10.5s //tensorflow/python/ops/numpy_ops:np_interop_test_cpu PASSED in 49.7s //tensorflow/python/ops/numpy_ops:np_logic_test_cpu PASSED in 18.9s //tensorflow/python/ops/numpy_ops:np_math_ops_test_cpu PASSED in 31.8s //tensorflow/python/ops/numpy_ops:np_random_test_cpu PASSED in 69.5s //tensorflow/python/ops/numpy_ops:np_utils_test_cpu PASSED in 10.1s //tensorflow/python/ops/numpy_ops/integration_test:np_config_test_cpu PASSED in 23.5s //tensorflow/python/ops/numpy_ops/integration_test:public_symbol_test PASSED in 29.3s //tensorflow/python/ops/parallel_for:array_test_cpu PASSED in 80.4s //tensorflow/python/ops/parallel_for:gradients_test_cpu PASSED in 15.0s //tensorflow/python/ops/parallel_for:xla_control_flow_ops_test_cpu PASSED in 75.9s //tensorflow/python/ops/ragged:convert_to_tensor_or_ragged_tensor_op_test PASSED in 10.2s //tensorflow/python/ops/ragged:ragged_batch_gather_op_test PASSED in 76.8s //tensorflow/python/ops/ragged:ragged_bincount_ops_test_cpu PASSED in 10.9s //tensorflow/python/ops/ragged:ragged_bitcast_op_test PASSED in 10.4s //tensorflow/python/ops/ragged:ragged_boolean_mask_op_test PASSED in 20.1s //tensorflow/python/ops/ragged:ragged_concat_op_test PASSED in 17.9s //tensorflow/python/ops/ragged:ragged_const_op_test PASSED in 34.3s //tensorflow/python/ops/ragged:ragged_constant_value_op_test PASSED in 10.2s //tensorflow/python/ops/ragged:ragged_cross_op_test PASSED in 36.7s //tensorflow/python/ops/ragged:ragged_dispatch_test PASSED in 137.0s //tensorflow/python/ops/ragged:ragged_dynamic_partition_op_test_cpu PASSED in 19.7s //tensorflow/python/ops/ragged:ragged_eager_test PASSED in 16.2s //tensorflow/python/ops/ragged:ragged_expand_dims_op_test PASSED in 10.8s //tensorflow/python/ops/ragged:ragged_factory_ops_test_cpu PASSED in 18.6s //tensorflow/python/ops/ragged:ragged_fill_empty_rows_op_test PASSED in 37.9s //tensorflow/python/ops/ragged:ragged_from_sparse_op_test PASSED in 10.9s //tensorflow/python/ops/ragged:ragged_from_tensor_op_test PASSED in 56.5s //tensorflow/python/ops/ragged:ragged_gather_nd_op_test PASSED in 17.3s //tensorflow/python/ops/ragged:ragged_map_flat_values_op_test PASSED in 11.8s //tensorflow/python/ops/ragged:ragged_map_fn_op_test PASSED in 18.4s //tensorflow/python/ops/ragged:ragged_math_ops_test PASSED in 30.9s //tensorflow/python/ops/ragged:ragged_matmul_op_test PASSED in 41.6s //tensorflow/python/ops/ragged:ragged_merge_dims_op_test PASSED in 47.9s //tensorflow/python/ops/ragged:ragged_one_hot_op_test PASSED in 12.7s //tensorflow/python/ops/ragged:ragged_operators_test PASSED in 26.2s //tensorflow/python/ops/ragged:ragged_placeholder_op_test PASSED in 10.8s //tensorflow/python/ops/ragged:ragged_print_op_test PASSED in 15.7s //tensorflow/python/ops/ragged:ragged_range_op_test PASSED in 9.9s //tensorflow/python/ops/ragged:ragged_rank_op_test PASSED in 11.6s //tensorflow/python/ops/ragged:ragged_reduce_op_test PASSED in 37.5s //tensorflow/python/ops/ragged:ragged_resize_image_op_test PASSED in 20.6s //tensorflow/python/ops/ragged:ragged_reverse_op_test PASSED in 15.7s //tensorflow/python/ops/ragged:ragged_row_lengths_op_test PASSED in 14.1s //tensorflow/python/ops/ragged:ragged_row_splits_to_segment_ids_op_test PASSED in 33.7s //tensorflow/python/ops/ragged:ragged_segment_ids_to_row_splits_op_test PASSED in 19.8s //tensorflow/python/ops/ragged:ragged_segment_op_test PASSED in 19.3s //tensorflow/python/ops/ragged:ragged_size_op_test PASSED in 10.4s //tensorflow/python/ops/ragged:ragged_split_op_test PASSED in 56.6s //tensorflow/python/ops/ragged:ragged_squeeze_op_test PASSED in 23.6s //tensorflow/python/ops/ragged:ragged_stack_op_test PASSED in 15.5s //tensorflow/python/ops/ragged:ragged_tensor_bounding_shape_op_test PASSED in 12.8s //tensorflow/python/ops/ragged:ragged_tensor_shape_test PASSED in 67.8s //tensorflow/python/ops/ragged:ragged_tile_op_test PASSED in 45.3s //tensorflow/python/ops/ragged:ragged_to_sparse_op_test PASSED in 12.1s //tensorflow/python/ops/ragged:ragged_to_tensor_op_test PASSED in 74.4s //tensorflow/python/ops/ragged:ragged_util_test PASSED in 25.0s //tensorflow/python/ops/ragged:ragged_where_op_test PASSED in 44.0s //tensorflow/python/ops/ragged:row_partition_test PASSED in 42.8s //tensorflow/python/ops/ragged:string_ngrams_op_test PASSED in 17.0s //tensorflow/python/ops/ragged:strings_reduce_join_op_test PASSED in 32.2s //tensorflow/python/ops/structured:structured_array_ops_test PASSED in 50.0s //tensorflow/python/ops/structured:structured_tensor_slice_test PASSED in 68.1s //tensorflow/python/ops/structured:structured_tensor_spec_test PASSED in 19.5s //tensorflow/python/ops/structured:structured_tensor_test PASSED in 56.2s //tensorflow/python/ops/v1_compat_tests:gradient_checker_test_cpu PASSED in 14.9s //tensorflow/python/platform:benchmark_test PASSED in 15.1s //tensorflow/python/platform:build_info_test PASSED in 10.4s //tensorflow/python/platform:resource_loader_test PASSED in 4.2s //tensorflow/python/profiler:pprof_profiler_test PASSED in 15.9s //tensorflow/python/profiler:profile_context_test_cpu PASSED in 27.0s //tensorflow/python/profiler:profiler_client_test_cpu PASSED in 11.2s //tensorflow/python/profiler:profiler_test_cpu PASSED in 19.7s //tensorflow/python/profiler:profiler_v2_test_cpu PASSED in 10.7s //tensorflow/python/profiler:profiler_wrapper_test PASSED in 13.5s //tensorflow/python/profiler:tfprof_logger_test PASSED in 11.1s //tensorflow/python/profiler/internal:flops_registry_test PASSED in 11.6s //tensorflow/python/profiler/internal:print_model_analysis_test PASSED in 11.4s //tensorflow/python/profiler/internal:run_metadata_test_cpu PASSED in 19.5s //tensorflow/python/saved_model:fingerprinting_test PASSED in 12.1s //tensorflow/python/saved_model:keras_injection_test PASSED in 23.5s //tensorflow/python/saved_model:load_v1_in_v2_test PASSED in 18.1s //tensorflow/python/saved_model:loader_test PASSED in 14.7s //tensorflow/python/saved_model:method_name_updater_test PASSED in 10.9s //tensorflow/python/saved_model:metrics_test PASSED in 12.0s //tensorflow/python/saved_model:nested_structure_coder_test PASSED in 11.2s //tensorflow/python/saved_model:pywrap_saved_model_fingerprinting_test PASSED in 9.2s //tensorflow/python/saved_model:pywrap_saved_model_metrics_test PASSED in 12.9s //tensorflow/python/saved_model:revived_types_test PASSED in 14.5s //tensorflow/python/saved_model:save_context_test PASSED in 9.0s //tensorflow/python/saved_model:save_test PASSED in 32.8s //tensorflow/python/saved_model:saved_model_test PASSED in 23.3s //tensorflow/python/saved_model:signature_def_utils_test PASSED in 9.8s //tensorflow/python/saved_model:simple_save_test PASSED in 10.1s //tensorflow/python/saved_model:tracing_utils_test PASSED in 10.8s //tensorflow/python/saved_model:utils_test PASSED in 21.4s //tensorflow/python/saved_model/model_utils:export_output_test PASSED in 10.7s //tensorflow/python/saved_model/model_utils:export_test PASSED in 17.7s //tensorflow/python/saved_model/model_utils:mode_keys_test PASSED in 9.7s //tensorflow/python/saved_model/registration:registration_saving_test PASSED in 31.3s //tensorflow/python/saved_model/registration:registration_test PASSED in 12.5s //tensorflow/python/saved_model/registration:tf_registration_test PASSED in 26.9s //tensorflow/python/saved_model/tests:variable_wrapper_test PASSED in 22.7s //tensorflow/python/summary:plugin_asset_test PASSED in 13.1s //tensorflow/python/summary:summary_iterator_test PASSED in 11.2s //tensorflow/python/summary:summary_test PASSED in 13.2s //tensorflow/python/summary:summary_v2_test PASSED in 23.9s //tensorflow/python/summary/writer:writer_test PASSED in 43.5s //tensorflow/python/tools:aot_compiled_test PASSED in 24.1s //tensorflow/python/tools:freeze_graph_test PASSED in 36.0s //tensorflow/python/tools:optimize_for_inference_test PASSED in 11.0s //tensorflow/python/tools:print_selective_registration_header_test PASSED in 41.6s //tensorflow/python/tools:saved_model_cli_test PASSED in 28.6s //tensorflow/python/tools:saved_model_utils_test PASSED in 9.7s //tensorflow/python/tools:strip_unused_test PASSED in 10.4s //tensorflow/python/tools/api/generator:create_python_api_test PASSED in 11.6s //tensorflow/python/tools/api/generator:output_init_files_test PASSED in 18.3s //tensorflow/python/tools/api/generator:tensorflow_doc_srcs_test PASSED in 14.9s //tensorflow/python/tpu:bfloat16_test PASSED in 10.3s //tensorflow/python/tpu:feature_column_test PASSED in 22.4s //tensorflow/python/tpu:topology_test PASSED in 11.0s //tensorflow/python/tpu:tpu_embedding_for_serving_test PASSED in 12.5s //tensorflow/python/tpu:tpu_embedding_v2_utils_test PASSED in 12.5s //tensorflow/python/tpu:tpu_infeed_test PASSED in 10.7s //tensorflow/python/tpu:tpu_sharding_test PASSED in 9.8s //tensorflow/python/tpu:tpu_test_wrapper_test PASSED in 9.3s //tensorflow/python/tpu/client:client_py_test PASSED in 13.9s //tensorflow/python/trackable:autotrackable_test PASSED in 11.6s //tensorflow/python/trackable:base_delegate_test PASSED in 12.2s //tensorflow/python/trackable:base_test PASSED in 11.4s //tensorflow/python/trackable:data_structures_test PASSED in 17.6s //tensorflow/python/trackable:python_state_test PASSED in 33.7s //tensorflow/python/trackable:resource_test PASSED in 13.3s //tensorflow/python/trackable:trackable_utils_test PASSED in 9.7s //tensorflow/python/training:adadelta_test_cpu PASSED in 36.8s //tensorflow/python/training:adagrad_da_test_cpu PASSED in 14.8s //tensorflow/python/training:adagrad_test_cpu PASSED in 17.7s //tensorflow/python/training:adam_test_cpu PASSED in 19.6s //tensorflow/python/training:basic_loops_test_cpu PASSED in 13.2s //tensorflow/python/training:basic_session_run_hooks_test PASSED in 28.8s //tensorflow/python/training:checkpoint_ops_test PASSED in 16.7s //tensorflow/python/training:coordinator_test_cpu PASSED in 19.1s //tensorflow/python/training:device_setter_test_cpu PASSED in 12.8s //tensorflow/python/training:ftrl_test_cpu PASSED in 18.3s //tensorflow/python/training:gradient_descent_test_cpu PASSED in 15.9s //tensorflow/python/training:input_test PASSED in 40.8s //tensorflow/python/training:momentum_test_cpu PASSED in 13.9s //tensorflow/python/training:monitored_session_test PASSED in 30.2s //tensorflow/python/training:moving_averages_test_cpu PASSED in 18.6s //tensorflow/python/training:optimizer_test_cpu PASSED in 16.1s //tensorflow/python/training:proximal_adagrad_test_cpu PASSED in 12.3s //tensorflow/python/training:proximal_gradient_descent_test_cpu PASSED in 11.2s //tensorflow/python/training:quantize_training_test_cpu PASSED in 15.7s //tensorflow/python/training:queue_runner_test_cpu PASSED in 12.3s //tensorflow/python/training:rmsprop_test_cpu PASSED in 22.4s //tensorflow/python/training:saver_large_partitioned_variable_test PASSED in 18.5s //tensorflow/python/training:saver_test_2gpu PASSED in 37.4s //tensorflow/python/training:saver_test_cpu PASSED in 54.8s //tensorflow/python/training:server_lib_multiple_containers_test PASSED in 17.5s //tensorflow/python/training:server_lib_same_variables_clear_container_test PASSED in 10.4s //tensorflow/python/training:server_lib_same_variables_clear_test PASSED in 12.5s //tensorflow/python/training:server_lib_same_variables_no_clear_test PASSED in 10.2s //tensorflow/python/training:server_lib_sparse_job_test PASSED in 10.5s //tensorflow/python/training:server_lib_test PASSED in 44.3s //tensorflow/python/training:session_manager_test_cpu PASSED in 118.0s //tensorflow/python/training:slot_creator_test_cpu PASSED in 11.9s //tensorflow/python/training:supervisor_test PASSED in 18.1s //tensorflow/python/training:training_ops_mlir_test_cpu PASSED in 13.9s //tensorflow/python/training:training_ops_test_cpu PASSED in 13.9s //tensorflow/python/training:training_util_test PASSED in 13.3s //tensorflow/python/training:warm_starting_util_test PASSED in 30.0s //tensorflow/python/training/experimental:loss_scale_optimizer_test PASSED in 21.8s //tensorflow/python/training/experimental:loss_scale_test PASSED in 28.0s //tensorflow/python/training/experimental:mixed_precision_test_cpu PASSED in 14.7s //tensorflow/python/training/saving:saveable_object_util_test PASSED in 15.2s //tensorflow/python/util:compat_test PASSED in 11.0s //tensorflow/python/util:decorator_utils_test PASSED in 11.1s //tensorflow/python/util:deprecation_test PASSED in 10.0s //tensorflow/python/util:dispatch_test PASSED in 35.8s //tensorflow/python/util:example_parser_configuration_test PASSED in 14.8s //tensorflow/python/util:fast_module_type_test PASSED in 12.2s //tensorflow/python/util:function_parameter_canonicalizer_test PASSED in 9.5s //tensorflow/python/util:function_utils_test PASSED in 11.3s //tensorflow/python/util:keyword_args_test PASSED in 10.7s //tensorflow/python/util:lazy_loader_test PASSED in 24.1s //tensorflow/python/util:lock_util_test PASSED in 13.0s //tensorflow/python/util:module_wrapper_test PASSED in 11.7s //tensorflow/python/util:nest_test PASSED in 30.0s //tensorflow/python/util:object_identity_test PASSED in 13.2s //tensorflow/python/util:pywrap_xla_ops_test PASSED in 2.9s //tensorflow/python/util:serialization_test PASSED in 12.0s //tensorflow/python/util:tf_contextlib_test PASSED in 10.0s //tensorflow/python/util:tf_decorator_test PASSED in 11.6s //tensorflow/python/util:tf_export_test PASSED in 34.0s //tensorflow/python/util:tf_inspect_test PASSED in 11.4s //tensorflow/python/util:tf_should_use_test PASSED in 12.9s //tensorflow/python/util:tf_stack_test PASSED in 13.9s //tensorflow/python/util:traceback_utils_test PASSED in 13.5s //tensorflow/python/util:type_annotations_test PASSED in 10.5s //tensorflow/python/util:variable_utils_test PASSED in 11.0s //tensorflow/python/util:vlog_test PASSED in 12.5s //tensorflow/tools/api/tests:module_test PASSED in 22.6s //tensorflow/tools/benchmark:benchmark_model_test PASSED in 2.0s //tensorflow/tools/common:public_api_test PASSED in 3.4s //tensorflow/tools/common:traverse_test PASSED in 2.7s //tensorflow/tools/compatibility:all_renames_v2_test PASSED in 11.6s //tensorflow/tools/compatibility:ast_edits_test PASSED in 13.6s //tensorflow/tools/compatibility:test_file_v1_0 PASSED in 21.5s //tensorflow/tools/compatibility:test_file_v2_0 PASSED in 32.7s //tensorflow/tools/compatibility:tf_upgrade_test PASSED in 9.5s //tensorflow/tools/compatibility:tf_upgrade_v2_safety_test PASSED in 9.8s //tensorflow/tools/docs:tf_doctest_test PASSED in 1.6s //tensorflow/tools/graph_transforms:file_utils_test PASSED in 0.9s //tensorflow/tools/graph_transforms:transform_graph_test PASSED in 1.9s //tensorflow/tools/graph_transforms:transform_utils_test PASSED in 2.5s //tensorflow/tools/graph_transforms:transforms_test PASSED in 8.3s //tensorflow/tools/proto_splitter:merge_test PASSED in 0.2s //tensorflow/tools/proto_splitter:split_graph_def_test PASSED in 9.8s //tensorflow/tools/proto_splitter:split_test PASSED in 13.2s //tensorflow/tools/proto_splitter:util_test PASSED in 10.8s //tensorflow/tools/proto_splitter/cc:composable_splitter_test PASSED in 0.2s //tensorflow/tools/proto_splitter/cc:graph_def_splitter_test PASSED in 0.6s //tensorflow/tools/proto_splitter/cc:saved_model_splitter_test PASSED in 0.2s //tensorflow/tools/proto_splitter/cc:util_test PASSED in 3.0s //tensorflow/tools/proto_splitter/python:saved_model_test PASSED in 11.0s //tensorflow/tools/proto_splitter/python:test_util_test PASSED in 10.7s //tensorflow/tools/proto_text:gen_proto_text_functions_lib_test PASSED in 0.7s //tensorflow/tools/tensorflow_builder/compat_checker:compat_checker_test PASSED in 1.0s //tensorflow/tsl/c:tsl_status_test PASSED in 0.1s //tensorflow/tsl/concurrency:async_value_ref_test PASSED in 0.6s //tensorflow/tsl/concurrency:async_value_test PASSED in 0.1s //tensorflow/tsl/concurrency:concurrent_vector_test PASSED in 0.1s //tensorflow/tsl/cuda:cudnn_version_test PASSED in 7.3s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_agent_test PASSED in 12.9s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_error_util_test PASSED in 0.2s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_recoverable_job_test PASSED in 0.2s //tensorflow/tsl/distributed_runtime/preemption:preemption_notifier_test PASSED in 5.4s //tensorflow/tsl/distributed_runtime/preemption:preemption_sync_manager_test PASSED in 5.6s //tensorflow/tsl/distributed_runtime/rpc:grpc_channel_test PASSED in 0.1s //tensorflow/tsl/distributed_runtime/rpc:grpc_util_test PASSED in 0.1s //tensorflow/tsl/framework:cancellation_test PASSED in 1.1s //tensorflow/tsl/framework:device_id_utils_test PASSED in 3.4s //tensorflow/tsl/framework/convolution:eigen_spatial_convolutions_test PASSED in 0.1s //tensorflow/tsl/lib/gtl:tsl_lib_gtl_tests PASSED in 0.2s //tensorflow/tsl/lib/hash:crc32c_test PASSED in 0.2s //tensorflow/tsl/lib/histogram:histogram_test PASSED in 0.4s //tensorflow/tsl/lib/io:buffered_file_test PASSED in 0.2s //tensorflow/tsl/lib/io:buffered_inputstream_test PASSED in 0.1s //tensorflow/tsl/lib/io:cache_test PASSED in 0.1s //tensorflow/tsl/lib/io:inputbuffer_test PASSED in 2.3s //tensorflow/tsl/lib/io:inputstream_interface_test PASSED in 0.4s //tensorflow/tsl/lib/io:random_inputstream_test PASSED in 0.2s //tensorflow/tsl/lib/io:record_reader_writer_test PASSED in 0.7s //tensorflow/tsl/lib/io:recordio_test PASSED in 0.3s //tensorflow/tsl/lib/io:table_test PASSED in 3.6s //tensorflow/tsl/lib/io:zlib_buffers_test PASSED in 23.0s //tensorflow/tsl/lib/io/snappy:snappy_test PASSED in 0.6s //tensorflow/tsl/lib/math:math_util_test PASSED in 0.1s //tensorflow/tsl/lib/random:distribution_sampler_test PASSED in 0.2s //tensorflow/tsl/lib/random:philox_random_test PASSED in 0.7s //tensorflow/tsl/lib/random:random_distributions_test PASSED in 19.3s //tensorflow/tsl/lib/random:simple_philox_test PASSED in 0.5s //tensorflow/tsl/lib/random:weighted_picker_test PASSED in 13.2s //tensorflow/tsl/platform:criticality_test PASSED in 0.1s //tensorflow/tsl/platform:ctstring_test PASSED in 0.2s //tensorflow/tsl/platform:denormal_test PASSED in 0.1s //tensorflow/tsl/platform:errors_test PASSED in 0.1s //tensorflow/tsl/platform:fingerprint_test PASSED in 1.2s //tensorflow/tsl/platform:hash_test PASSED in 0.1s //tensorflow/tsl/platform:integral_types_test PASSED in 0.4s //tensorflow/tsl/platform:intrusive_ptr_test PASSED in 0.2s //tensorflow/tsl/platform:logging_test PASSED in 35.5s //tensorflow/tsl/platform:mutex_test PASSED in 0.3s //tensorflow/tsl/platform:net_test PASSED in 0.1s //tensorflow/tsl/platform:numbers_test PASSED in 0.3s //tensorflow/tsl/platform:path_test PASSED in 0.1s //tensorflow/tsl/platform:port_test PASSED in 8.2s //tensorflow/tsl/platform:random_test PASSED in 1.7s //tensorflow/tsl/platform:refcount_test PASSED in 0.2s //tensorflow/tsl/platform:retrying_file_system_test PASSED in 0.6s //tensorflow/tsl/platform:retrying_utils_test PASSED in 0.1s //tensorflow/tsl/platform:scanner_test PASSED in 0.6s //tensorflow/tsl/platform:setround_test PASSED in 0.1s //tensorflow/tsl/platform:stacktrace_handler_test PASSED in 1.3s //tensorflow/tsl/platform:stacktrace_test PASSED in 0.1s //tensorflow/tsl/platform:status_matchers_test PASSED in 0.1s //tensorflow/tsl/platform:status_test PASSED in 0.2s //tensorflow/tsl/platform:statusor_test PASSED in 2.5s //tensorflow/tsl/platform:str_util_test PASSED in 0.1s //tensorflow/tsl/platform:strcat_test PASSED in 0.3s //tensorflow/tsl/platform:stringpiece_test PASSED in 0.1s //tensorflow/tsl/platform:stringprintf_test PASSED in 0.1s //tensorflow/tsl/platform:subprocess_test PASSED in 0.7s //tensorflow/tsl/platform:tstring_test PASSED in 0.1s //tensorflow/tsl/platform:unbounded_work_queue_test PASSED in 0.9s //tensorflow/tsl/platform/cloud:compute_engine_metadata_client_test PASSED in 0.2s //tensorflow/tsl/platform/cloud:compute_engine_zone_provider_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:curl_http_request_test PASSED in 8.6s //tensorflow/tsl/platform/cloud:expiring_lru_cache_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:gcs_dns_cache_test PASSED in 0.5s //tensorflow/tsl/platform/cloud:gcs_file_system_test PASSED in 5.9s //tensorflow/tsl/platform/cloud:gcs_throttle_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:google_auth_provider_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:oauth_client_test PASSED in 0.2s //tensorflow/tsl/platform/cloud:ram_file_block_cache_test PASSED in 2.3s //tensorflow/tsl/platform/cloud:time_util_test PASSED in 0.1s //tensorflow/tsl/profiler/backends/cpu:traceme_recorder_test PASSED in 0.2s //tensorflow/tsl/profiler/convert:trace_container_test PASSED in 0.6s //tensorflow/tsl/profiler/convert:trace_events_to_json_test PASSED in 0.3s //tensorflow/tsl/profiler/convert:xla_op_utils_test PASSED in 0.2s //tensorflow/tsl/profiler/convert:xplane_to_trace_events_test PASSED in 0.1s //tensorflow/tsl/profiler/lib:profiler_factory_test PASSED in 0.8s //tensorflow/tsl/profiler/lib:profiler_lock_test PASSED in 0.1s //tensorflow/tsl/profiler/lib:scoped_annotation_test PASSED in 0.1s //tensorflow/tsl/profiler/lib:traceme_encode_test PASSED in 0.1s //tensorflow/tsl/profiler/rpc/client:profiler_client_test PASSED in 3.4s //tensorflow/tsl/profiler/rpc/client:remote_profiler_session_manager_test PASSED in 3.5s //tensorflow/tsl/profiler/utils:buffer_pool_test PASSED in 1.6s //tensorflow/tsl/profiler/utils:group_events_test PASSED in 0.5s //tensorflow/tsl/profiler/utils:parse_annotation_test PASSED in 0.2s //tensorflow/tsl/profiler/utils:preprocess_xplane_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:tf_op_utils_test PASSED in 0.2s //tensorflow/tsl/profiler/utils:timespan_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:tpu_xplane_utils_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:xplane_builder_test PASSED in 0.3s //tensorflow/tsl/profiler/utils:xplane_utils_test PASSED in 0.4s //tensorflow/tsl/util:device_name_utils_test PASSED in 0.1s //tensorflow/tsl/util:stats_calculator_test PASSED in 0.1s //tensorflow/compiler/tests:complex_div_test_cpu PASSED in 9.2s Stats over 2 runs: max = 9.2s, min = 8.8s, avg = 9.0s, dev = 0.2s //tensorflow/compiler/tests:complex_div_test_cpu_mlir_bridge_test PASSED in 28.4s Stats over 2 runs: max = 28.4s, min = 27.8s, avg = 28.1s, dev = 0.3s //tensorflow/compiler/xla/tests:conditional_test_cpu PASSED in 9.9s Stats over 2 runs: max = 9.9s, min = 8.7s, avg = 9.3s, dev = 0.6s //tensorflow/python/data/experimental/kernel_tests/optimization:optimization_test PASSED in 22.7s Stats over 2 runs: max = 22.7s, min = 18.0s, avg = 20.4s, dev = 2.3s //tensorflow/python/data/experimental/kernel_tests/service:metadata_test PASSED in 25.1s Stats over 2 runs: max = 25.1s, min = 23.8s, avg = 24.5s, dev = 0.7s //tensorflow/python/data/kernel_tests:padded_batch_test PASSED in 42.8s Stats over 2 runs: max = 42.8s, min = 41.9s, avg = 42.4s, dev = 0.4s //tensorflow/python/data/kernel_tests:repeat_test PASSED in 68.9s Stats over 2 runs: max = 68.9s, min = 68.1s, avg = 68.5s, dev = 0.4s //tensorflow/python/data/kernel_tests:window_test PASSED in 62.5s Stats over 2 runs: max = 62.5s, min = 50.1s, avg = 56.3s, dev = 6.2s //tensorflow/python/kernel_tests/array_ops:scatter_nd_ops_test_cpu PASSED in 16.6s Stats over 2 runs: max = 16.6s, min = 15.4s, avg = 16.0s, dev = 0.6s //tensorflow/python/kernel_tests/control_flow:functional_ops_test_cpu PASSED in 32.1s Stats over 2 runs: max = 32.1s, min = 30.5s, avg = 31.3s, dev = 0.8s //tensorflow/python/kernel_tests/control_flow:map_fn_test_cpu PASSED in 12.8s Stats over 2 runs: max = 12.8s, min = 11.3s, avg = 12.0s, dev = 0.8s //tensorflow/python/kernel_tests/nn_ops:atrous_conv2d_test_cpu PASSED in 28.8s Stats over 2 runs: max = 28.8s, min = 18.4s, avg = 23.6s, dev = 5.2s //tensorflow/python/kernel_tests/nn_ops:bias_op_d9m_test_cpu PASSED in 119.3s Stats over 2 runs: max = 119.3s, min = 49.6s, avg = 84.4s, dev = 34.9s //tensorflow/python/kernel_tests/nn_ops:conv2d_backprop_filter_grad_test_cpu PASSED in 10.3s Stats over 2 runs: max = 10.3s, min = 9.9s, avg = 10.1s, dev = 0.2s //tensorflow/python/ops:control_flow_ops_test_cpu PASSED in 30.6s Stats over 2 runs: max = 30.6s, min = 26.5s, avg = 28.6s, dev = 2.1s //tensorflow/compiler/tests:spacetobatch_op_test_cpu PASSED in 11.8s Stats over 3 runs: max = 11.8s, min = 11.5s, avg = 11.7s, dev = 0.1s //tensorflow/compiler/tests:spacetobatch_op_test_cpu_mlir_bridge_test PASSED in 13.0s Stats over 3 runs: max = 13.0s, min = 12.8s, avg = 12.9s, dev = 0.1s //tensorflow/compiler/xla/tests:triangular_solve_test_cpu PASSED in 54.4s Stats over 3 runs: max = 54.4s, min = 52.0s, avg = 53.6s, dev = 1.1s //tensorflow/core/data/service:thread_safe_buffer_test PASSED in 0.1s Stats over 3 runs: max = 0.1s, min = 0.1s, avg = 0.1s, dev = 0.0s //tensorflow/python/data/experimental/kernel_tests/service:multi_process_cluster_test PASSED in 24.7s Stats over 3 runs: max = 24.7s, min = 14.4s, avg = 21.3s, dev = 4.8s //tensorflow/python/data/kernel_tests:unique_test PASSED in 16.3s Stats over 3 runs: max = 16.3s, min = 13.3s, avg = 14.5s, dev = 1.3s //tensorflow/python/distribute/coordinator:metric_utils_test PASSED in 22.0s Stats over 3 runs: max = 22.0s, min = 17.0s, avg = 19.9s, dev = 2.1s //tensorflow/python/kernel_tests/array_ops:gather_op_test_cpu PASSED in 47.8s Stats over 3 runs: max = 47.8s, min = 29.7s, avg = 36.6s, dev = 7.9s //tensorflow/python/kernel_tests/array_ops:weights_broadcast_test PASSED in 25.5s Stats over 3 runs: max = 25.5s, min = 24.4s, avg = 24.8s, dev = 0.5s //tensorflow/python/kernel_tests/distributions:util_test_cpu PASSED in 15.6s Stats over 3 runs: max = 15.6s, min = 14.5s, avg = 15.1s, dev = 0.5s //tensorflow/python/kernel_tests/linalg:matrix_triangular_solve_op_test_cpu PASSED in 130.3s Stats over 3 runs: max = 130.3s, min = 10.4s, avg = 50.4s, dev = 56.5s //tensorflow/python/kernel_tests/random:multinomial_op_big_test_cpu PASSED in 22.7s Stats over 3 runs: max = 22.7s, min = 19.2s, avg = 20.5s, dev = 1.6s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:collective_ops.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/collective_ops.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/collective_ops.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/collective_ops.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:fft.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.2s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/fft.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/fft.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/fft.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:legalize_i1_vector_transfers.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.2s, avg = 0.3s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/legalize_i1_vector_transfers.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/legalize_i1_vector_transfers.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/legalize_i1_vector_transfers.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:library_ops_to_cpu_runtime.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/library_ops_to_cpu_runtime.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/library_ops_to_cpu_runtime.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/library_ops_to_cpu_runtime.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:lmhlo_custom_call.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/lmhlo_custom_call.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/lmhlo_custom_call.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/lmhlo_custom_call.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:remove_copies_to_out_params.mlir.test FAILED in 3 out of 3 in 1.0s Stats over 3 runs: max = 1.0s, min = 0.1s, avg = 0.4s, dev = 0.4s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/remove_copies_to_out_params.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/remove_copies_to_out_params.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/remove_copies_to_out_params.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:rng_bit_generator.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/rng_bit_generator.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/rng_bit_generator.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/rng_bit_generator.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_abi_legalization.mlir.test FAILED in 3 out of 3 in 0.4s Stats over 3 runs: max = 0.4s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_abi_legalization.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_abi_legalization.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_abi_legalization.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_infeed.mlir.test FAILED in 3 out of 3 in 0.1s Stats over 3 runs: max = 0.1s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_infeed.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_infeed.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_infeed.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_memref_element_cast_to_llvm.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_memref_element_cast_to_llvm.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_memref_element_cast_to_llvm.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_memref_element_cast_to_llvm.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_outfeed.mlir.test FAILED in 3 out of 3 in 0.4s Stats over 3 runs: max = 0.4s, min = 0.1s, avg = 0.3s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_outfeed.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_outfeed.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests/xla_cpu_outfeed.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_concurrent_regions.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/add_concurrent_regions.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/add_concurrent_regions.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/add_concurrent_regions.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_hlo_trace.mlir.test FAILED in 3 out of 3 in 0.7s Stats over 3 runs: max = 0.7s, min = 0.1s, avg = 0.4s, dev = 0.2s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/add_hlo_trace.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/add_hlo_trace.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/add_hlo_trace.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_launch.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_launch.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_launch.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_launch.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memcpy.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_memcpy.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_memcpy.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_memcpy.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memset.mlir.test FAILED in 3 out of 3 in 0.4s Stats over 3 runs: max = 0.4s, min = 0.2s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_memset.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_memset.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/gpu_memset.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_case.mlir.test FAILED in 3 out of 3 in 0.5s Stats over 3 runs: max = 0.5s, min = 0.1s, avg = 0.2s, dev = 0.2s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_case.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_case.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_case.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_custom_call.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_custom_call.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_custom_call.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_custom_call.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_fft.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_fft.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_fft.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_fft.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cholesky.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.2s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_cholesky.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_cholesky.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_cholesky.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_conv.mlir.test FAILED in 3 out of 3 in 0.1s Stats over 3 runs: max = 0.1s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_conv.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_conv.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_conv.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cublas_lt_matmul.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_cublas_lt_matmul.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_cublas_lt_matmul.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_cublas_lt_matmul.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_gemm.mlir.test FAILED in 3 out of 3 in 0.1s Stats over 3 runs: max = 0.1s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_gemm.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_gemm.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_gpu_gemm.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_infeed.mlir.test FAILED in 3 out of 3 in 0.5s Stats over 3 runs: max = 0.5s, min = 0.3s, avg = 0.4s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_infeed.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_infeed.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_infeed.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_outfeed.mlir.test FAILED in 3 out of 3 in 0.1s Stats over 3 runs: max = 0.1s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_outfeed.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_outfeed.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_outfeed.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_send_recv.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_send_recv.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_send_recv.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_send_recv.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_while.mlir.test FAILED in 3 out of 3 in 0.5s Stats over 3 runs: max = 0.5s, min = 0.2s, avg = 0.4s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_while.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_while.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/lmhlo_while.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:memref_get_global_to_arg.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/memref_get_global_to_arg.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/memref_get_global_to_arg.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/memref_get_global_to_arg.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:outline_cuda_graphs.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.2s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/outline_cuda_graphs.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/outline_cuda_graphs.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/outline_cuda_graphs.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:stream_assignment.mlir.test FAILED in 3 out of 3 in 0.6s Stats over 3 runs: max = 0.6s, min = 0.1s, avg = 0.4s, dev = 0.2s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/stream_assignment.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/stream_assignment.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests/stream_assignment.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/framework/tests:legalize-xla-framework.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.2s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/framework/tests/legalize-xla-framework.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/framework/tests/legalize-xla-framework.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/framework/tests/legalize-xla-framework.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/framework/tests:outline-with-xla-framework.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/framework/tests/outline-with-xla-framework.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/framework/tests/outline-with-xla-framework.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/framework/tests/outline-with-xla-framework.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/framework/tests:xla-framework.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.2s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/framework/tests/xla-framework.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/framework/tests/xla-framework.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/framework/tests/xla-framework.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/math/transforms/tests:math_optimization.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/math/transforms/tests/math_optimization.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/math/transforms/tests/math_optimization.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/math/transforms/tests/math_optimization.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/memref/transforms/tests:aligned_allocations.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.2s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/memref/transforms/tests/aligned_allocations.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/memref/transforms/tests/aligned_allocations.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/memref/transforms/tests/aligned_allocations.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops.mlir.test FAILED in 3 out of 3 in 0.5s Stats over 3 runs: max = 0.5s, min = 0.1s, avg = 0.3s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/ir/tests/ops.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/ir/tests/ops.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/ir/tests/ops.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops_verify.mlir.test FAILED in 3 out of 3 in 0.1s Stats over 3 runs: max = 0.1s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/ir/tests/ops_verify.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/ir/tests/ops_verify.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/ir/tests/ops_verify.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/runtime/ir/tests:testlib.mlir.test FAILED in 3 out of 3 in 0.1s Stats over 3 runs: max = 0.1s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/ir/tests/testlib.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/ir/tests/testlib.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/ir/tests/testlib.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/runtime/transforms/tests:compilation_pipeline.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/compilation_pipeline.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/compilation_pipeline.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/compilation_pipeline.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_asserts.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/convert_asserts.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/convert_asserts.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/convert_asserts.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_custom_calls.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/convert_custom_calls.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/convert_custom_calls.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/convert_custom_calls.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/runtime/transforms/tests:export_functions.mlir.test FAILED in 3 out of 3 in 0.4s Stats over 3 runs: max = 0.4s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/export_functions.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/export_functions.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/export_functions.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/runtime/transforms/tests:ordinal_assignment.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/ordinal_assignment.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/ordinal_assignment.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/ordinal_assignment.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/runtime/transforms/tests:rt_to_llvm.mlir.test FAILED in 3 out of 3 in 0.4s Stats over 3 runs: max = 0.4s, min = 0.2s, avg = 0.3s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/rt_to_llvm.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/rt_to_llvm.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/runtime/transforms/tests/rt_to_llvm.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:erase-op-without-results.mlir.test FAILED in 3 out of 3 in 0.4s Stats over 3 runs: max = 0.4s, min = 0.1s, avg = 0.3s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/erase-op-without-results.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/erase-op-without-results.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/erase-op-without-results.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:inline-scf-while.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/inline-scf-while.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/inline-scf-while.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/inline-scf-while.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:reduce-scf-forall-bounds.mlir.test FAILED in 3 out of 3 in 0.5s Stats over 3 runs: max = 0.5s, min = 0.1s, avg = 0.2s, dev = 0.2s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/reduce-scf-forall-bounds.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/reduce-scf-forall-bounds.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/reduce-scf-forall-bounds.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-constant.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.1s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-op-with-constant.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-op-with-constant.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-op-with-constant.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-value.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-op-with-value.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-op-with-value.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-op-with-value.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-operand-with-constant.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-operand-with-constant.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-operand-with-constant.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/replace-operand-with-constant.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:return-operands-of-terminator-operands.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/return-operands-of-terminator-operands.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/return-operands-of-terminator-operands.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/return-operands-of-terminator-operands.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:truncate-function.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.2s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/truncate-function.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/truncate-function.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests/truncate-function.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:bisect.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/bisect.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/bisect.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/bisect.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:no-bug.mlir.test FAILED in 3 out of 3 in 0.1s Stats over 3 runs: max = 0.1s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/no-bug.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/no-bug.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/no-bug.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:snapshot.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/snapshot.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/snapshot.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests/snapshot.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/xla_cpu/tests:bufferize.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/xla_cpu/tests/bufferize.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/xla_cpu/tests/bufferize.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/xla_cpu/tests/bufferize.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/xla_cpu/tests:invalid.mlir.test FAILED in 3 out of 3 in 0.9s Stats over 3 runs: max = 0.9s, min = 0.9s, avg = 0.9s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/xla_cpu/tests/invalid.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/xla_cpu/tests/invalid.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/xla_cpu/tests/invalid.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/mlir/xla_cpu/tests:ops.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/xla_cpu/tests/ops.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/xla_cpu/tests/ops.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/mlir/xla_cpu/tests/ops.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/python/ifrt/ir/tests:ifrt_duplicated_callee_elimination.mlir.test FAILED in 3 out of 3 in 0.4s Stats over 3 runs: max = 0.4s, min = 0.2s, avg = 0.3s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/ifrt_duplicated_callee_elimination.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/ifrt_duplicated_callee_elimination.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/ifrt_duplicated_callee_elimination.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/python/ifrt/ir/tests:spmd_interface_verification.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/spmd_interface_verification.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/spmd_interface_verification.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/spmd_interface_verification.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_array.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.2s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_array.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_array.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_array.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_assemble.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_assemble.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_assemble.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_assemble.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_attrs.mlir.test FAILED in 3 out of 3 in 0.5s Stats over 3 runs: max = 0.5s, min = 0.4s, avg = 0.4s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_attrs.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_attrs.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_attrs.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_call.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_call.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_call.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call_loaded_executable.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_call_loaded_executable.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_call_loaded_executable.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_call_loaded_executable.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_disassemble.mlir.test FAILED in 3 out of 3 in 0.8s Stats over 3 runs: max = 0.8s, min = 0.3s, avg = 0.5s, dev = 0.2s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_disassemble.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_disassemble.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_disassemble.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_loaded_executable.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_loaded_executable.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_loaded_executable.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_loaded_executable.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_reshard.mlir.test FAILED in 3 out of 3 in 0.9s Stats over 3 runs: max = 0.9s, min = 0.6s, avg = 0.7s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_reshard.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_reshard.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/python/ifrt/ir/tests/verify_reshard.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:bool_compare.hlotxt.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/bool_compare.hlotxt.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/bool_compare.hlotxt.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/bool_compare.hlotxt.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:case_conditional.hlotxt.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/case_conditional.hlotxt.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/case_conditional.hlotxt.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/case_conditional.hlotxt.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:dynamic_param.hlo.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.2s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/dynamic_param.hlo.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/dynamic_param.hlo.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/dynamic_param.hlo.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:entry_computation_layout.hlotxt.test FAILED in 3 out of 3 in 0.4s Stats over 3 runs: max = 0.4s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/entry_computation_layout.hlotxt.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/entry_computation_layout.hlotxt.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/entry_computation_layout.hlotxt.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:frontend_attributes.hlotxt.test FAILED in 3 out of 3 in 0.1s Stats over 3 runs: max = 0.1s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/frontend_attributes.hlotxt.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/frontend_attributes.hlotxt.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/frontend_attributes.hlotxt.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fully_connected_reference_model.hlotxt.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.2s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/fully_connected_reference_model.hlotxt.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/fully_connected_reference_model.hlotxt.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/fully_connected_reference_model.hlotxt.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fusion.hlotxt.test FAILED in 3 out of 3 in 0.9s Stats over 3 runs: max = 0.9s, min = 0.1s, avg = 0.4s, dev = 0.3s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/fusion.hlotxt.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/fusion.hlotxt.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/fusion.hlotxt.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:if_conditional.hlotxt.test FAILED in 3 out of 3 in 0.4s Stats over 3 runs: max = 0.4s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/if_conditional.hlotxt.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/if_conditional.hlotxt.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/if_conditional.hlotxt.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import.hlotxt.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/import.hlotxt.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/import.hlotxt.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/import.hlotxt.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import_async.hlotxt.test FAILED in 3 out of 3 in 0.1s Stats over 3 runs: max = 0.1s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/import_async.hlotxt.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/import_async.hlotxt.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/import_async.hlotxt.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:layouts_and_names.hlotxt.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/layouts_and_names.hlotxt.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/layouts_and_names.hlotxt.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/layouts_and_names.hlotxt.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:location.hlotxt.test FAILED in 3 out of 3 in 0.8s Stats over 3 runs: max = 0.8s, min = 0.2s, avg = 0.4s, dev = 0.3s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/location.hlotxt.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/location.hlotxt.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/location.hlotxt.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:module_attributes.hlo.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/module_attributes.hlo.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/module_attributes.hlo.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/module_attributes.hlo.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:send_recv.hlotxt.test FAILED in 3 out of 3 in 0.4s Stats over 3 runs: max = 0.4s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/send_recv.hlotxt.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/send_recv.hlotxt.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/send_recv.hlotxt.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:simple.hlo.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.2s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/simple.hlo.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/simple.hlo.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/simple.hlo.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:spmd_module_sharding.hlo.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/spmd_module_sharding.hlo.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/spmd_module_sharding.hlo.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/spmd_module_sharding.hlo.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:stacktrace_to_location.hlo.test FAILED in 3 out of 3 in 0.6s Stats over 3 runs: max = 0.6s, min = 0.6s, avg = 0.6s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/stacktrace_to_location.hlo.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/stacktrace_to_location.hlo.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/stacktrace_to_location.hlo.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:types.hlotxt.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/types.hlotxt.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/types.hlotxt.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/types.hlotxt.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:while.hlotxt.test FAILED in 3 out of 3 in 0.5s Stats over 3 runs: max = 0.5s, min = 0.3s, avg = 0.4s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/while.hlotxt.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/while.hlotxt.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/hlo_to_mhlo/tests/while.hlotxt.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:add.mlir.test FAILED in 3 out of 3 in 0.5s Stats over 3 runs: max = 0.5s, min = 0.1s, avg = 0.3s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/add.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/add.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/add.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:case.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/case.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/case.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/case.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:dynamic.mlir.test FAILED in 3 out of 3 in 0.5s Stats over 3 runs: max = 0.5s, min = 0.2s, avg = 0.3s, dev = 0.2s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/dynamic.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/dynamic.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/dynamic.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export-with-layouts.mlir.test FAILED in 3 out of 3 in 0.4s Stats over 3 runs: max = 0.4s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export-with-layouts.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export-with-layouts.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export-with-layouts.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.2s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_and_check_layouts.mlir.test FAILED in 3 out of 3 in 0.5s Stats over 3 runs: max = 0.5s, min = 0.1s, avg = 0.3s, dev = 0.2s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_and_check_layouts.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_and_check_layouts.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_and_check_layouts.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_large_constants.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.2s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_large_constants.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_large_constants.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_large_constants.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_replicas.mlir.test FAILED in 3 out of 3 in 0.8s Stats over 3 runs: max = 0.8s, min = 0.3s, avg = 0.4s, dev = 0.2s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_replicas.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_replicas.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/export_replicas.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:frontend_attributes.mlir.test FAILED in 3 out of 3 in 0.5s Stats over 3 runs: max = 0.5s, min = 0.3s, avg = 0.4s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/frontend_attributes.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/frontend_attributes.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/frontend_attributes.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:fusion.mlir.test FAILED in 3 out of 3 in 0.8s Stats over 3 runs: max = 0.8s, min = 0.4s, avg = 0.6s, dev = 0.2s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/fusion.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/fusion.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/fusion.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:if.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/if.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/if.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/if.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:input_output_aliasing.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/input_output_aliasing.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/input_output_aliasing.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/input_output_aliasing.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:layouts_and_names.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/layouts_and_names.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/layouts_and_names.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/layouts_and_names.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_op_metadata.mlir.test FAILED in 3 out of 3 in 0.5s Stats over 3 runs: max = 0.5s, min = 0.1s, avg = 0.3s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/location_to_op_metadata.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/location_to_op_metadata.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/location_to_op_metadata.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_stacktrace.mlir.test FAILED in 3 out of 3 in 0.4s Stats over 3 runs: max = 0.4s, min = 0.1s, avg = 0.3s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/location_to_stacktrace.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/location_to_stacktrace.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/location_to_stacktrace.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:missing_main.mlir.test FAILED in 3 out of 3 in 0.6s Stats over 3 runs: max = 0.6s, min = 0.2s, avg = 0.4s, dev = 0.2s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/missing_main.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/missing_main.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/missing_main.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:module_attributes.mlir.test FAILED in 3 out of 3 in 1.3s Stats over 3 runs: max = 1.3s, min = 0.1s, avg = 0.6s, dev = 0.5s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/module_attributes.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/module_attributes.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/module_attributes.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:multiple_return_tuple.mlir.test FAILED in 3 out of 3 in 0.4s Stats over 3 runs: max = 0.4s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/multiple_return_tuple.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/multiple_return_tuple.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/multiple_return_tuple.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:opaque_elements_attr.mlir.test FAILED in 3 out of 3 in 0.6s Stats over 3 runs: max = 0.6s, min = 0.2s, avg = 0.4s, dev = 0.2s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/opaque_elements_attr.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/opaque_elements_attr.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/opaque_elements_attr.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:rng_get_and_update_state.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/rng_get_and_update_state.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/rng_get_and_update_state.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/rng_get_and_update_state.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:sharding.mlir.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/sharding.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/sharding.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/sharding.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:simple.mlir.test FAILED in 3 out of 3 in 1.1s Stats over 3 runs: max = 1.1s, min = 0.1s, avg = 0.5s, dev = 0.4s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/simple.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/simple.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/simple.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:unsupported_type.mlir.test FAILED in 3 out of 3 in 0.4s Stats over 3 runs: max = 0.4s, min = 0.2s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/unsupported_type.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/unsupported_type.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/unsupported_type.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:while.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/while.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/while.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_hlo/tests/while.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:hlo_text_to_lhlo_no_opt.hlotxt.test FAILED in 3 out of 3 in 0.3s Stats over 3 runs: max = 0.3s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/hlo_text_to_lhlo_no_opt.hlotxt.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/hlo_text_to_lhlo_no_opt.hlotxt.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/hlo_text_to_lhlo_no_opt.hlotxt.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:no_opt_ops.hlotxt.test FAILED in 3 out of 3 in 0.1s Stats over 3 runs: max = 0.1s, min = 0.1s, avg = 0.1s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/no_opt_ops.hlotxt.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/no_opt_ops.hlotxt.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/no_opt_ops.hlotxt.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:non_identity_layouts.hlotxt.test FAILED in 3 out of 3 in 0.4s Stats over 3 runs: max = 0.4s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/non_identity_layouts.hlotxt.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/non_identity_layouts.hlotxt.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/non_identity_layouts.hlotxt.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:ops.mlir.test FAILED in 3 out of 3 in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.2s, dev = 0.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/ops.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/ops.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/ops.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:passthrough.mlir.test FAILED in 3 out of 3 in 0.5s Stats over 3 runs: max = 0.5s, min = 0.5s, avg = 0.5s, dev = 0.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/passthrough.mlir.test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/passthrough.mlir.test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests/passthrough.mlir.test/test_attempts/attempt_2.log //tensorflow/compiler/xla/tests:dynamic_ops_test_cpu PASSED in 10.0s Stats over 4 runs: max = 10.0s, min = 8.5s, avg = 9.1s, dev = 0.6s //tensorflow/core/kernels:example_parsing_ops_test PASSED in 1.6s Stats over 4 runs: max = 1.6s, min = 1.2s, avg = 1.4s, dev = 0.2s //tensorflow/python/data/experimental/kernel_tests:auto_shard_dataset_test PASSED in 34.7s Stats over 4 runs: max = 34.7s, min = 20.4s, avg = 27.4s, dev = 5.5s //tensorflow/python/data/experimental/kernel_tests:map_and_batch_test PASSED in 48.2s Stats over 4 runs: max = 48.2s, min = 26.3s, avg = 32.3s, dev = 9.2s //tensorflow/python/data/experimental/kernel_tests:parse_example_dataset_test PASSED in 30.3s Stats over 4 runs: max = 30.3s, min = 17.9s, avg = 23.7s, dev = 5.4s //tensorflow/python/data/experimental/kernel_tests:rebatch_dataset_test PASSED in 41.2s Stats over 4 runs: max = 41.2s, min = 25.1s, avg = 31.1s, dev = 6.3s //tensorflow/python/data/experimental/kernel_tests:sql_dataset_test PASSED in 77.6s Stats over 4 runs: max = 77.6s, min = 61.3s, avg = 69.4s, dev = 8.1s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_ft_test PASSED in 25.3s Stats over 4 runs: max = 25.3s, min = 23.3s, avg = 24.0s, dev = 0.8s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_test PASSED in 61.1s Stats over 4 runs: max = 61.1s, min = 31.8s, avg = 46.3s, dev = 12.9s //tensorflow/python/data/kernel_tests:batch_test PASSED in 35.5s Stats over 4 runs: max = 35.5s, min = 28.0s, avg = 30.2s, dev = 3.1s //tensorflow/python/data/kernel_tests:fixed_length_record_dataset_test PASSED in 19.3s Stats over 4 runs: max = 19.3s, min = 11.9s, avg = 15.5s, dev = 3.6s //tensorflow/python/data/kernel_tests:from_generator_test PASSED in 22.4s Stats over 4 runs: max = 22.4s, min = 13.9s, avg = 17.5s, dev = 3.1s //tensorflow/python/data/kernel_tests:group_by_window_test PASSED in 23.6s Stats over 4 runs: max = 23.6s, min = 10.9s, avg = 16.5s, dev = 5.6s //tensorflow/python/data/kernel_tests:ragged_batch_test PASSED in 22.1s Stats over 4 runs: max = 22.1s, min = 19.6s, avg = 20.8s, dev = 1.2s //tensorflow/python/data/kernel_tests:skip_test PASSED in 40.7s Stats over 4 runs: max = 40.7s, min = 33.2s, avg = 36.9s, dev = 3.6s //tensorflow/python/data/kernel_tests:take_test PASSED in 39.3s Stats over 4 runs: max = 39.3s, min = 38.6s, avg = 38.9s, dev = 0.3s //tensorflow/python/data/kernel_tests:take_while_test PASSED in 45.2s Stats over 4 runs: max = 45.2s, min = 41.4s, avg = 43.5s, dev = 1.6s //tensorflow/python/data/kernel_tests:text_line_dataset_test PASSED in 22.5s Stats over 4 runs: max = 22.5s, min = 16.0s, avg = 19.4s, dev = 2.7s //tensorflow/python/data/kernel_tests:zip_test PASSED in 17.4s Stats over 4 runs: max = 17.4s, min = 16.0s, avg = 16.7s, dev = 0.5s //tensorflow/python/debug/lib:dumping_callback_test_cpu PASSED in 19.8s Stats over 4 runs: max = 19.8s, min = 17.5s, avg = 18.5s, dev = 0.8s //tensorflow/python/distribute:cross_device_ops_test_2gpu PASSED in 50.7s Stats over 4 runs: max = 50.7s, min = 40.5s, avg = 45.1s, dev = 4.6s //tensorflow/python/distribute:cross_device_ops_test_cpu PASSED in 33.0s Stats over 4 runs: max = 33.0s, min = 23.9s, avg = 28.3s, dev = 3.5s //tensorflow/python/framework:convert_to_constants_test PASSED in 44.1s Stats over 4 runs: max = 44.1s, min = 35.5s, avg = 39.1s, dev = 3.2s //tensorflow/python/kernel_tests:collective_ops_test_2gpu PASSED in 38.8s Stats over 4 runs: max = 38.8s, min = 36.6s, avg = 37.3s, dev = 0.9s //tensorflow/python/kernel_tests:collective_ops_test_cpu PASSED in 39.5s Stats over 4 runs: max = 39.5s, min = 36.4s, avg = 38.2s, dev = 1.1s //tensorflow/python/kernel_tests/array_ops:concat_op_test_cpu PASSED in 16.7s Stats over 4 runs: max = 16.7s, min = 12.9s, avg = 15.0s, dev = 1.4s //tensorflow/python/kernel_tests/array_ops:init_ops_test_cpu PASSED in 69.6s Stats over 4 runs: max = 69.6s, min = 30.9s, avg = 50.8s, dev = 18.2s //tensorflow/python/kernel_tests/array_ops:split_op_test_cpu PASSED in 32.2s Stats over 4 runs: max = 32.2s, min = 12.6s, avg = 20.0s, dev = 8.0s //tensorflow/python/kernel_tests/linalg:einsum_op_test_cpu PASSED in 85.8s Stats over 4 runs: max = 85.8s, min = 17.6s, avg = 45.0s, dev = 27.6s //tensorflow/python/kernel_tests/linalg:linear_operator_lower_triangular_test_cpu PASSED in 34.6s Stats over 4 runs: max = 34.6s, min = 31.2s, avg = 32.7s, dev = 1.5s //tensorflow/python/kernel_tests/nn_ops:conv_ops_test_cpu PASSED in 40.5s Stats over 4 runs: max = 40.5s, min = 34.8s, avg = 37.5s, dev = 2.3s //tensorflow/python/kernel_tests/random:random_gamma_test_cpu PASSED in 113.5s Stats over 4 runs: max = 113.5s, min = 11.5s, avg = 56.2s, dev = 43.4s //tensorflow/python/kernel_tests/signal:window_ops_test_cpu PASSED in 26.1s Stats over 4 runs: max = 26.1s, min = 24.8s, avg = 25.4s, dev = 0.6s //tensorflow/python/ops:nn_batchnorm_test_cpu PASSED in 22.2s Stats over 4 runs: max = 22.2s, min = 18.0s, avg = 19.3s, dev = 1.6s //tensorflow/python/ops:nn_fused_batchnorm_d9m_test_cpu PASSED in 34.8s Stats over 4 runs: max = 34.8s, min = 33.3s, avg = 34.1s, dev = 0.5s //tensorflow/python/ops/ragged:ragged_gather_op_test PASSED in 90.8s Stats over 4 runs: max = 90.8s, min = 38.8s, avg = 63.5s, dev = 18.5s //tensorflow/python/ops/ragged:ragged_getitem_test PASSED in 58.6s Stats over 4 runs: max = 58.6s, min = 52.3s, avg = 54.5s, dev = 2.5s //tensorflow/compiler/tests:async_comp_test_cpu PASSED in 14.0s Stats over 5 runs: max = 14.0s, min = 11.8s, avg = 12.5s, dev = 0.8s //tensorflow/compiler/tests:conv3d_test_cpu PASSED in 24.3s Stats over 5 runs: max = 24.3s, min = 12.2s, avg = 16.9s, dev = 4.2s //tensorflow/compiler/tests:conv3d_test_cpu_mlir_bridge_test PASSED in 15.6s Stats over 5 runs: max = 15.6s, min = 9.5s, avg = 12.3s, dev = 2.6s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu PASSED in 16.4s Stats over 5 runs: max = 16.4s, min = 11.0s, avg = 13.5s, dev = 2.3s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu_mlir_bridge_test PASSED in 17.4s Stats over 5 runs: max = 17.4s, min = 11.9s, avg = 14.5s, dev = 2.1s //tensorflow/compiler/tests:fused_batchnorm_test_cpu PASSED in 10.5s Stats over 5 runs: max = 10.5s, min = 9.6s, avg = 10.1s, dev = 0.3s //tensorflow/compiler/tests:fused_batchnorm_test_cpu_mlir_bridge_test PASSED in 13.3s Stats over 5 runs: max = 13.3s, min = 12.7s, avg = 13.1s, dev = 0.2s //tensorflow/compiler/tests:image_ops_jit_compile_test_cpu PASSED in 13.0s Stats over 5 runs: max = 13.0s, min = 9.7s, avg = 10.4s, dev = 1.3s //tensorflow/compiler/tests:reduce_ops_test_cpu PASSED in 13.9s Stats over 5 runs: max = 13.9s, min = 10.7s, avg = 12.5s, dev = 1.3s //tensorflow/compiler/tests:reduce_ops_test_cpu_mlir_bridge_test PASSED in 16.6s Stats over 5 runs: max = 16.6s, min = 11.2s, avg = 14.0s, dev = 2.2s //tensorflow/compiler/tests:repeat_op_test_cpu PASSED in 11.2s Stats over 5 runs: max = 11.2s, min = 9.1s, avg = 9.8s, dev = 0.7s //tensorflow/compiler/tests:repeat_op_test_cpu_mlir_bridge_test PASSED in 13.5s Stats over 5 runs: max = 13.5s, min = 11.6s, avg = 12.1s, dev = 0.7s //tensorflow/compiler/tests:special_math_test_cpu PASSED in 104.3s Stats over 5 runs: max = 104.3s, min = 17.1s, avg = 50.1s, dev = 29.8s //tensorflow/compiler/tests:special_math_test_cpu_mlir_bridge_test PASSED in 101.6s Stats over 5 runs: max = 101.6s, min = 20.5s, avg = 56.8s, dev = 28.0s //tensorflow/compiler/xla/client/lib:self_adjoint_eig_test_cpu PASSED in 27.1s Stats over 5 runs: max = 27.1s, min = 13.6s, avg = 21.7s, dev = 6.0s //tensorflow/core/grappler/optimizers:constant_folding_test PASSED in 2.9s Stats over 5 runs: max = 2.9s, min = 1.6s, avg = 2.2s, dev = 0.6s //tensorflow/dtensor/python/tests:layout_propagation_test_cpu PASSED in 13.5s Stats over 5 runs: max = 13.5s, min = 13.0s, avg = 13.2s, dev = 0.2s //tensorflow/dtensor/python/tests:multi_mesh_test_cpu PASSED in 12.1s Stats over 5 runs: max = 12.1s, min = 9.4s, avg = 10.9s, dev = 0.9s //tensorflow/python/distribute:mirrored_strategy_test_2gpu PASSED in 13.2s Stats over 5 runs: max = 13.2s, min = 12.5s, avg = 12.9s, dev = 0.2s //tensorflow/python/distribute:mirrored_strategy_test_cpu PASSED in 19.3s Stats over 5 runs: max = 19.3s, min = 18.1s, avg = 18.7s, dev = 0.5s //tensorflow/python/distribute:moving_averages_test_2gpu PASSED in 28.4s Stats over 5 runs: max = 28.4s, min = 23.1s, avg = 25.3s, dev = 2.0s //tensorflow/python/distribute:moving_averages_test_cpu PASSED in 23.4s Stats over 5 runs: max = 23.4s, min = 20.1s, avg = 21.6s, dev = 1.1s //tensorflow/python/distribute:vars_test_2gpu PASSED in 26.5s Stats over 5 runs: max = 26.5s, min = 20.5s, avg = 23.5s, dev = 2.3s //tensorflow/python/distribute:vars_test_cpu PASSED in 33.8s Stats over 5 runs: max = 33.8s, min = 18.8s, avg = 21.9s, dev = 6.0s //tensorflow/python/eager:device_placement_test_cpu PASSED in 25.8s Stats over 5 runs: max = 25.8s, min = 24.0s, avg = 25.0s, dev = 0.6s //tensorflow/python/eager:forwardprop_test_cpu PASSED in 121.7s Stats over 5 runs: max = 121.7s, min = 18.9s, avg = 58.7s, dev = 34.3s //tensorflow/python/eager/polymorphic_function:gradients_test_cpu PASSED in 23.6s Stats over 5 runs: max = 23.6s, min = 17.6s, avg = 21.0s, dev = 2.3s //tensorflow/python/kernel_tests/linalg:cholesky_op_test_cpu PASSED in 44.1s Stats over 5 runs: max = 44.1s, min = 32.8s, avg = 38.1s, dev = 3.9s //tensorflow/python/kernel_tests/linalg:linear_operator_adjoint_test_cpu PASSED in 43.5s Stats over 5 runs: max = 43.5s, min = 39.6s, avg = 42.5s, dev = 1.5s //tensorflow/python/kernel_tests/linalg:linear_operator_composition_test_cpu PASSED in 54.9s Stats over 5 runs: max = 54.9s, min = 53.6s, avg = 54.6s, dev = 0.5s //tensorflow/python/kernel_tests/linalg:linear_operator_diag_test_cpu PASSED in 27.8s Stats over 5 runs: max = 27.8s, min = 25.6s, avg = 26.5s, dev = 0.8s //tensorflow/python/kernel_tests/linalg:linear_operator_full_matrix_test_cpu PASSED in 33.4s Stats over 5 runs: max = 33.4s, min = 30.9s, avg = 32.1s, dev = 1.0s //tensorflow/python/kernel_tests/linalg:linear_operator_householder_test_cpu PASSED in 38.6s Stats over 5 runs: max = 38.6s, min = 37.3s, avg = 38.0s, dev = 0.5s //tensorflow/python/kernel_tests/linalg:linear_operator_identity_test_cpu PASSED in 52.0s Stats over 5 runs: max = 52.0s, min = 48.5s, avg = 49.7s, dev = 1.2s //tensorflow/python/kernel_tests/linalg:linear_operator_inversion_test_cpu PASSED in 36.0s Stats over 5 runs: max = 36.0s, min = 31.9s, avg = 33.8s, dev = 1.3s //tensorflow/python/kernel_tests/linalg:linear_operator_permutation_test_cpu PASSED in 30.5s Stats over 5 runs: max = 30.5s, min = 29.0s, avg = 29.8s, dev = 0.6s //tensorflow/python/kernel_tests/linalg:linear_operator_toeplitz_test_cpu PASSED in 46.9s Stats over 5 runs: max = 46.9s, min = 46.0s, avg = 46.5s, dev = 0.4s //tensorflow/python/kernel_tests/linalg:linear_operator_tridiag_test_cpu PASSED in 124.9s Stats over 5 runs: max = 124.9s, min = 113.2s, avg = 119.4s, dev = 4.0s //tensorflow/python/kernel_tests/linalg:linear_operator_util_test_cpu PASSED in 10.8s Stats over 5 runs: max = 10.8s, min = 3.9s, avg = 6.7s, dev = 3.0s //tensorflow/python/kernel_tests/linalg:linear_operator_zeros_test_cpu PASSED in 16.9s Stats over 5 runs: max = 16.9s, min = 15.9s, avg = 16.3s, dev = 0.3s //tensorflow/python/kernel_tests/nn_ops:fractional_avg_pool_op_test PASSED in 22.0s Stats over 5 runs: max = 22.0s, min = 10.6s, avg = 13.4s, dev = 4.4s //tensorflow/python/kernel_tests/nn_ops:fractional_max_pool_op_test PASSED in 16.2s Stats over 5 runs: max = 16.2s, min = 10.0s, avg = 11.6s, dev = 2.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_ops_test_cpu PASSED in 31.7s Stats over 5 runs: max = 31.7s, min = 12.3s, avg = 17.2s, dev = 7.3s //tensorflow/python/ops/parallel_for:math_test_cpu PASSED in 126.3s Stats over 5 runs: max = 126.3s, min = 48.1s, avg = 79.1s, dev = 28.2s //tensorflow/compiler/tests:scan_ops_test_cpu PASSED in 15.9s Stats over 6 runs: max = 15.9s, min = 12.1s, avg = 14.0s, dev = 1.2s //tensorflow/compiler/tests:scan_ops_test_cpu_mlir_bridge_test PASSED in 22.9s Stats over 6 runs: max = 22.9s, min = 16.0s, avg = 19.8s, dev = 2.6s //tensorflow/python/data/experimental/kernel_tests:make_batched_features_dataset_test PASSED in 28.6s Stats over 6 runs: max = 28.6s, min = 10.2s, avg = 17.9s, dev = 7.6s //tensorflow/python/kernel_tests/array_ops:diag_op_test_cpu PASSED in 45.8s Stats over 6 runs: max = 45.8s, min = 11.1s, avg = 18.5s, dev = 12.3s //tensorflow/python/kernel_tests/math_ops:reduction_ops_test_cpu PASSED in 49.3s Stats over 6 runs: max = 49.3s, min = 23.9s, avg = 35.7s, dev = 8.0s //tensorflow/python/ops:accumulate_n_benchmark_cpu PASSED in 10.1s Stats over 6 runs: max = 10.1s, min = 9.8s, avg = 9.9s, dev = 0.1s //tensorflow/python/distribute/experimental/rpc:rpc_ops_test PASSED in 15.5s Stats over 7 runs: max = 15.5s, min = 9.7s, avg = 12.1s, dev = 2.3s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu PASSED in 57.8s Stats over 8 runs: max = 57.8s, min = 8.9s, avg = 24.4s, dev = 16.1s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu_mlir_bridge_test PASSED in 71.9s Stats over 8 runs: max = 71.9s, min = 8.1s, avg = 29.1s, dev = 21.7s //tensorflow/dtensor/python/tests:input_util_test PASSED in 29.5s Stats over 8 runs: max = 29.5s, min = 17.9s, avg = 24.3s, dev = 3.4s //tensorflow/python/data/experimental/kernel_tests:csv_dataset_test PASSED in 31.8s Stats over 8 runs: max = 31.8s, min = 10.6s, avg = 17.9s, dev = 7.5s //tensorflow/python/data/experimental/kernel_tests:parallel_interleave_test PASSED in 31.6s Stats over 8 runs: max = 31.6s, min = 15.4s, avg = 23.0s, dev = 5.6s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_ft_test PASSED in 39.5s Stats over 8 runs: max = 39.5s, min = 9.0s, avg = 23.5s, dev = 12.9s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_test PASSED in 39.8s Stats over 8 runs: max = 39.8s, min = 12.3s, avg = 19.6s, dev = 9.5s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_test PASSED in 25.6s Stats over 8 runs: max = 25.6s, min = 10.7s, avg = 16.0s, dev = 5.6s //tensorflow/python/data/experimental/kernel_tests/service:fault_tolerance_test PASSED in 17.9s Stats over 8 runs: max = 17.9s, min = 11.0s, avg = 13.0s, dev = 2.5s //tensorflow/python/data/kernel_tests:filter_test PASSED in 18.5s Stats over 8 runs: max = 18.5s, min = 14.3s, avg = 16.6s, dev = 1.2s //tensorflow/python/data/kernel_tests:flat_map_test PASSED in 27.4s Stats over 8 runs: max = 27.4s, min = 16.6s, avg = 21.3s, dev = 3.8s //tensorflow/python/data/kernel_tests:shard_test PASSED in 25.5s Stats over 8 runs: max = 25.5s, min = 17.0s, avg = 20.9s, dev = 2.7s //tensorflow/python/data/kernel_tests:shuffle_test PASSED in 80.5s Stats over 8 runs: max = 80.5s, min = 42.1s, avg = 49.3s, dev = 11.8s //tensorflow/python/data/kernel_tests:tf_record_dataset_test PASSED in 29.0s Stats over 8 runs: max = 29.0s, min = 17.7s, avg = 23.1s, dev = 3.0s //tensorflow/python/distribute/failure_handling:gce_failure_handler_test PASSED in 81.3s Stats over 8 runs: max = 81.3s, min = 20.1s, avg = 40.1s, dev = 23.7s //tensorflow/python/kernel_tests/linalg:linalg_ops_test_cpu PASSED in 55.5s Stats over 8 runs: max = 55.5s, min = 36.0s, avg = 46.2s, dev = 7.1s //tensorflow/python/kernel_tests/linalg:linear_operator_block_diag_test_cpu PASSED in 75.5s Stats over 8 runs: max = 75.5s, min = 55.8s, avg = 67.1s, dev = 6.6s //tensorflow/python/kernel_tests/linalg:linear_operator_block_lower_triangular_test_cpu PASSED in 70.8s Stats over 8 runs: max = 70.8s, min = 39.3s, avg = 55.4s, dev = 10.6s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_d9m_test_cpu PASSED in 63.1s Stats over 8 runs: max = 63.1s, min = 5.0s, avg = 14.8s, dev = 18.5s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_test_cpu PASSED in 13.3s Stats over 8 runs: max = 13.3s, min = 11.1s, avg = 12.6s, dev = 0.7s //tensorflow/python/kernel_tests/signal:fft_ops_test_cpu PASSED in 20.7s Stats over 8 runs: max = 20.7s, min = 11.0s, avg = 15.3s, dev = 4.3s //tensorflow/python/ops/ragged:dynamic_ragged_shape_test PASSED in 50.8s Stats over 8 runs: max = 50.8s, min = 33.4s, avg = 40.1s, dev = 6.3s //tensorflow/python/ops/ragged:ragged_tensor_test PASSED in 25.4s Stats over 8 runs: max = 25.4s, min = 15.4s, avg = 18.7s, dev = 3.0s //tensorflow/compiler/tests:bincount_op_test_cpu PASSED in 9.2s Stats over 10 runs: max = 9.2s, min = 6.1s, avg = 8.0s, dev = 1.1s //tensorflow/compiler/tests:conv2d_test_cpu PASSED in 11.5s Stats over 10 runs: max = 11.5s, min = 5.4s, avg = 10.3s, dev = 1.7s //tensorflow/compiler/tests:conv2d_test_cpu_mlir_bridge_test PASSED in 10.4s Stats over 10 runs: max = 10.4s, min = 4.6s, avg = 6.9s, dev = 2.1s //tensorflow/compiler/tests:random_ops_test_cpu PASSED in 17.5s Stats over 10 runs: max = 17.5s, min = 7.8s, avg = 13.3s, dev = 2.8s //tensorflow/compiler/tests:random_ops_test_cpu_mlir_bridge_test PASSED in 20.6s Stats over 10 runs: max = 20.6s, min = 5.2s, avg = 11.7s, dev = 4.9s //tensorflow/compiler/tests:stateless_random_ops_test_cpu PASSED in 75.4s Stats over 10 runs: max = 75.4s, min = 44.9s, avg = 58.9s, dev = 9.9s //tensorflow/compiler/tests:stateless_random_ops_test_cpu_mlir_bridge_test PASSED in 84.9s Stats over 10 runs: max = 84.9s, min = 47.4s, avg = 63.4s, dev = 11.7s //tensorflow/compiler/tests:stochastic_cast_op_test_cpu PASSED in 23.1s Stats over 10 runs: max = 23.1s, min = 20.3s, avg = 22.2s, dev = 0.8s //tensorflow/compiler/xla/client/lib:svd_test_cpu PASSED in 30.9s Stats over 10 runs: max = 30.9s, min = 6.1s, avg = 13.8s, dev = 8.8s //tensorflow/compiler/xla/client/lib:tridiagonal_test_cpu PASSED in 8.0s Stats over 10 runs: max = 8.0s, min = 6.6s, avg = 7.2s, dev = 0.4s //tensorflow/compiler/xla/service/cpu:cpu_runtime_test PASSED in 11.6s Stats over 10 runs: max = 11.6s, min = 0.6s, avg = 7.9s, dev = 3.7s //tensorflow/python/data/kernel_tests:rejection_resample_test PASSED in 32.2s Stats over 10 runs: max = 32.2s, min = 21.4s, avg = 25.7s, dev = 3.4s //tensorflow/python/distribute:input_lib_type_spec_test_2gpu PASSED in 22.4s Stats over 10 runs: max = 22.4s, min = 7.4s, avg = 15.9s, dev = 5.3s //tensorflow/python/distribute:input_lib_type_spec_test_cpu PASSED in 23.1s Stats over 10 runs: max = 23.1s, min = 13.0s, avg = 17.7s, dev = 3.5s //tensorflow/python/framework:config_vgpu_test_2gpu PASSED in 10.5s Stats over 10 runs: max = 10.5s, min = 8.9s, avg = 9.9s, dev = 0.6s //tensorflow/python/framework:config_vgpu_test_cpu PASSED in 11.9s Stats over 10 runs: max = 11.9s, min = 9.0s, avg = 10.2s, dev = 1.1s //tensorflow/python/framework:function_test_cpu PASSED in 51.9s Stats over 10 runs: max = 51.9s, min = 8.9s, avg = 15.8s, dev = 12.9s //tensorflow/python/grappler:cluster_test_cpu PASSED in 10.5s Stats over 10 runs: max = 10.5s, min = 7.4s, avg = 9.1s, dev = 1.1s //tensorflow/python/kernel_tests/array_ops:array_ops_test_cpu PASSED in 17.1s Stats over 10 runs: max = 17.1s, min = 11.6s, avg = 13.9s, dev = 1.7s //tensorflow/python/kernel_tests/array_ops:inplace_ops_test_cpu PASSED in 12.3s Stats over 10 runs: max = 12.3s, min = 11.1s, avg = 11.6s, dev = 0.4s //tensorflow/python/kernel_tests/data_structures:tensor_array_ops_test_cpu PASSED in 15.2s Stats over 10 runs: max = 15.2s, min = 11.0s, avg = 12.5s, dev = 1.4s //tensorflow/python/kernel_tests/linalg:linear_operator_low_rank_update_test_cpu PASSED in 91.4s Stats over 10 runs: max = 91.4s, min = 85.8s, avg = 89.1s, dev = 1.6s //tensorflow/python/kernel_tests/linalg:tridiagonal_matmul_op_test_cpu PASSED in 142.1s Stats over 10 runs: max = 142.1s, min = 9.3s, avg = 23.2s, dev = 39.6s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_ops_test_cpu PASSED in 73.8s Stats over 10 runs: max = 73.8s, min = 45.8s, avg = 56.5s, dev = 8.6s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_test_cpu PASSED in 28.0s Stats over 10 runs: max = 28.0s, min = 8.1s, avg = 16.6s, dev = 7.8s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_test_cpu PASSED in 23.9s Stats over 10 runs: max = 23.9s, min = 10.1s, avg = 13.6s, dev = 5.0s //tensorflow/python/kernel_tests/nn_ops:rnn_test_cpu PASSED in 15.2s Stats over 10 runs: max = 15.2s, min = 13.5s, avg = 14.2s, dev = 0.5s //tensorflow/python/kernel_tests/random:random_index_shuffle_test PASSED in 12.0s Stats over 10 runs: max = 12.0s, min = 10.5s, avg = 11.2s, dev = 0.5s //tensorflow/python/kernel_tests/random:stateless_random_ops_test_cpu PASSED in 112.3s Stats over 10 runs: max = 112.3s, min = 19.7s, avg = 64.1s, dev = 42.7s //tensorflow/python/ops:special_math_ops_test_cpu PASSED in 40.7s Stats over 10 runs: max = 40.7s, min = 6.6s, avg = 14.5s, dev = 9.2s //tensorflow/python/ops:weak_tensor_special_math_ops_test_cpu PASSED in 14.9s Stats over 10 runs: max = 14.9s, min = 11.4s, avg = 12.9s, dev = 1.0s //tensorflow/python/ops/numpy_ops/tests:np_indexing_test PASSED in 105.0s Stats over 10 runs: max = 105.0s, min = 90.7s, avg = 98.8s, dev = 5.2s //tensorflow/python/ops/ragged:ragged_tensor_supported_values_test PASSED in 36.8s Stats over 10 runs: max = 36.8s, min = 30.6s, avg = 32.5s, dev = 2.2s //tensorflow/python/saved_model:load_test_cpu PASSED in 60.3s Stats over 10 runs: max = 60.3s, min = 34.8s, avg = 40.8s, dev = 6.9s //tensorflow/python/distribute/failure_handling:failure_handler_test FLAKY, failed in 2 out of 10 in 356.8s Stats over 10 runs: max = 356.8s, min = 46.5s, avg = 101.9s, dev = 85.9s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/failure_handling/failure_handler_test/shard_5_of_8/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/failure_handling/failure_handler_test/shard_1_of_8/test_attempts/attempt_1.log //tensorflow/compiler/tests:fft_test_cpu PASSED in 24.1s Stats over 12 runs: max = 24.1s, min = 10.5s, avg = 17.5s, dev = 5.2s //tensorflow/compiler/xla/service:triangular_solve_expander_test PASSED in 4.6s Stats over 12 runs: max = 4.6s, min = 2.1s, avg = 3.2s, dev = 0.8s //tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 24.7s Stats over 12 runs: max = 24.7s, min = 11.8s, avg = 15.9s, dev = 4.1s //tensorflow/python/data/kernel_tests:choose_from_datasets_test PASSED in 21.8s Stats over 12 runs: max = 21.8s, min = 14.5s, avg = 17.6s, dev = 2.1s //tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu PASSED in 13.6s Stats over 12 runs: max = 13.6s, min = 5.6s, avg = 10.3s, dev = 2.1s //tensorflow/python/distribute:multi_process_runner_test_2gpu PASSED in 225.3s Stats over 12 runs: max = 225.3s, min = 13.9s, avg = 52.9s, dev = 58.1s //tensorflow/python/distribute:multi_process_runner_test_cpu PASSED in 240.7s Stats over 12 runs: max = 240.7s, min = 26.9s, avg = 65.5s, dev = 58.8s //tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 23.4s Stats over 15 runs: max = 23.4s, min = 17.0s, avg = 19.3s, dev = 1.7s //tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu PASSED in 50.6s Stats over 15 runs: max = 50.6s, min = 8.2s, avg = 18.4s, dev = 9.4s //tensorflow/compiler/tests:ftrl_test_cpu PASSED in 16.5s Stats over 16 runs: max = 16.5s, min = 10.6s, avg = 13.0s, dev = 1.9s //tensorflow/compiler/tests:ternary_ops_test_cpu PASSED in 14.0s Stats over 16 runs: max = 14.0s, min = 9.1s, avg = 10.8s, dev = 1.3s //tensorflow/compiler/tests:ternary_ops_test_cpu_mlir_bridge_test PASSED in 19.2s Stats over 16 runs: max = 19.2s, min = 11.7s, avg = 13.7s, dev = 1.8s //tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 21.0s Stats over 16 runs: max = 21.0s, min = 13.2s, avg = 17.0s, dev = 2.6s //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test PASSED in 33.6s Stats over 16 runs: max = 33.6s, min = 9.5s, avg = 19.0s, dev = 6.7s //tensorflow/python/data/kernel_tests:snapshot_test PASSED in 44.4s Stats over 16 runs: max = 44.4s, min = 15.2s, avg = 33.4s, dev = 6.2s //tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 27.2s Stats over 16 runs: max = 27.2s, min = 6.2s, avg = 10.2s, dev = 5.0s //tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test PASSED in 16.8s Stats over 16 runs: max = 16.8s, min = 13.4s, avg = 14.5s, dev = 0.7s //tensorflow/python/kernel_tests/signal:dct_ops_test_cpu PASSED in 17.4s Stats over 16 runs: max = 17.4s, min = 14.1s, avg = 15.4s, dev = 0.8s //tensorflow/python/ops:image_ops_test_cpu PASSED in 24.6s Stats over 16 runs: max = 24.6s, min = 12.7s, avg = 17.2s, dev = 3.1s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test PASSED in 67.2s Stats over 17 runs: max = 67.2s, min = 7.6s, avg = 24.5s, dev = 22.9s //tensorflow/python/data/kernel_tests:map_test PASSED in 40.4s Stats over 19 runs: max = 40.4s, min = 14.3s, avg = 23.4s, dev = 5.7s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu PASSED in 9.8s Stats over 20 runs: max = 9.8s, min = 4.3s, avg = 6.8s, dev = 2.0s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test PASSED in 10.4s Stats over 20 runs: max = 10.4s, min = 8.3s, avg = 9.3s, dev = 0.6s //tensorflow/compiler/tests:pooling_ops_test_cpu PASSED in 13.6s Stats over 20 runs: max = 13.6s, min = 7.9s, avg = 9.3s, dev = 1.3s //tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test PASSED in 15.3s Stats over 20 runs: max = 15.3s, min = 6.8s, avg = 9.5s, dev = 1.8s //tensorflow/compiler/xla/tests:convolution_dimension_numbers_test_cpu PASSED in 7.6s Stats over 20 runs: max = 7.6s, min = 6.4s, avg = 7.0s, dev = 0.4s //tensorflow/compiler/xla/tests:dot_operation_single_threaded_runtime_test_cpu PASSED in 12.8s Stats over 20 runs: max = 12.8s, min = 10.0s, avg = 11.0s, dev = 0.7s //tensorflow/compiler/xla/tests:dot_operation_test_cpu PASSED in 12.7s Stats over 20 runs: max = 12.7s, min = 10.2s, avg = 11.2s, dev = 0.7s //tensorflow/compiler/xla/tests:prng_test_cpu PASSED in 8.2s Stats over 20 runs: max = 8.2s, min = 6.4s, avg = 7.0s, dev = 0.5s //tensorflow/compiler/xla/tests:reduce_window_test_cpu PASSED in 47.1s Stats over 20 runs: max = 47.1s, min = 7.1s, avg = 19.7s, dev = 14.7s //tensorflow/python/autograph/tests:loop_control_flow_test PASSED in 26.4s Stats over 20 runs: max = 26.4s, min = 15.7s, avg = 22.8s, dev = 3.1s //tensorflow/python/kernel_tests:metrics_test PASSED in 41.9s Stats over 20 runs: max = 41.9s, min = 9.4s, avg = 21.4s, dev = 10.1s //tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu PASSED in 14.4s Stats over 20 runs: max = 14.4s, min = 9.4s, avg = 11.1s, dev = 1.4s //tensorflow/python/kernel_tests/data_structures:barrier_ops_test PASSED in 15.9s Stats over 20 runs: max = 15.9s, min = 7.4s, avg = 10.5s, dev = 2.2s //tensorflow/python/kernel_tests/linalg:eig_op_test PASSED in 50.0s Stats over 20 runs: max = 50.0s, min = 9.8s, avg = 20.2s, dev = 13.6s //tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu PASSED in 129.2s Stats over 20 runs: max = 129.2s, min = 36.7s, avg = 64.3s, dev = 24.0s //tensorflow/python/kernel_tests/linalg:norm_op_test_cpu PASSED in 10.9s Stats over 20 runs: max = 10.9s, min = 6.9s, avg = 8.8s, dev = 1.1s //tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu PASSED in 16.1s Stats over 20 runs: max = 16.1s, min = 10.0s, avg = 12.7s, dev = 1.8s //tensorflow/python/kernel_tests/linalg:qr_op_test_cpu PASSED in 161.3s Stats over 20 runs: max = 161.3s, min = 41.7s, avg = 86.4s, dev = 38.4s //tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu PASSED in 29.9s Stats over 20 runs: max = 29.9s, min = 9.1s, avg = 15.4s, dev = 6.1s //tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu PASSED in 23.3s Stats over 20 runs: max = 23.3s, min = 8.7s, avg = 15.4s, dev = 5.0s //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu PASSED in 42.8s Stats over 20 runs: max = 42.8s, min = 18.9s, avg = 35.9s, dev = 7.9s //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu PASSED in 67.6s Stats over 20 runs: max = 67.6s, min = 7.8s, avg = 29.0s, dev = 21.0s //tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu PASSED in 31.5s Stats over 20 runs: max = 31.5s, min = 22.4s, avg = 24.6s, dev = 2.0s //tensorflow/python/data/experimental/kernel_tests/service:local_workers_test PASSED in 26.8s Stats over 24 runs: max = 26.8s, min = 11.3s, avg = 18.9s, dev = 4.0s //tensorflow/python/data/kernel_tests:interleave_test PASSED in 23.9s Stats over 24 runs: max = 23.9s, min = 7.9s, avg = 15.3s, dev = 4.8s //tensorflow/python/data/kernel_tests:sample_from_datasets_test PASSED in 18.5s Stats over 24 runs: max = 18.5s, min = 4.2s, avg = 9.8s, dev = 4.4s //tensorflow/compiler/xla/tests:array_elementwise_ops_test_cpu PASSED in 8.5s Stats over 25 runs: max = 8.5s, min = 6.9s, avg = 7.8s, dev = 0.5s //tensorflow/compiler/xla/tests:select_and_scatter_test_cpu PASSED in 35.8s Stats over 25 runs: max = 35.8s, min = 6.9s, avg = 11.4s, dev = 7.2s //tensorflow/compiler/xla/tests:convolution_variants_test_cpu PASSED in 21.4s Stats over 30 runs: max = 21.4s, min = 6.5s, avg = 10.0s, dev = 3.3s //tensorflow/compiler/xla/tests:iota_test_cpu PASSED in 14.7s Stats over 30 runs: max = 14.7s, min = 12.0s, avg = 13.0s, dev = 0.7s //tensorflow/compiler/xla/tests:params_test_cpu PASSED in 7.6s Stats over 30 runs: max = 7.6s, min = 6.2s, avg = 6.7s, dev = 0.4s //tensorflow/compiler/xla/tests:reshape_test_cpu PASSED in 8.3s Stats over 30 runs: max = 8.3s, min = 6.1s, avg = 7.0s, dev = 0.6s //tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu PASSED in 27.7s Stats over 30 runs: max = 27.7s, min = 3.8s, avg = 11.4s, dev = 7.6s //tensorflow/compiler/xla/tests:reduce_test_cpu PASSED in 8.9s Stats over 31 runs: max = 8.9s, min = 6.9s, avg = 7.6s, dev = 0.5s //tensorflow/compiler/xla/tests:scalar_computations_test_cpu PASSED in 8.7s Stats over 32 runs: max = 8.7s, min = 6.5s, avg = 7.4s, dev = 0.7s //tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 25.6s Stats over 32 runs: max = 25.6s, min = 7.0s, avg = 12.8s, dev = 4.3s //tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 44.0s Stats over 32 runs: max = 44.0s, min = 31.2s, avg = 38.6s, dev = 3.1s //tensorflow/compiler/xla/tests:batch_normalization_test_cpu PASSED in 9.6s Stats over 40 runs: max = 9.6s, min = 7.7s, avg = 8.7s, dev = 0.5s //tensorflow/compiler/xla/tests:bfloat16_test_cpu PASSED in 10.3s Stats over 40 runs: max = 10.3s, min = 8.4s, avg = 9.5s, dev = 0.5s //tensorflow/compiler/xla/tests:conv_depthwise_backprop_filter_test_cpu PASSED in 16.8s Stats over 40 runs: max = 16.8s, min = 9.1s, avg = 12.1s, dev = 1.6s //tensorflow/compiler/xla/tests:slice_test_cpu PASSED in 15.7s Stats over 40 runs: max = 15.7s, min = 8.4s, avg = 10.4s, dev = 1.8s //tensorflow/core/kernels:stochastic_cast_op_test PASSED in 2.4s Stats over 48 runs: max = 2.4s, min = 0.6s, avg = 1.1s, dev = 0.4s //tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 71.5s Stats over 50 runs: max = 71.5s, min = 28.7s, avg = 47.7s, dev = 13.4s //tensorflow/compiler/tests:sort_ops_test_cpu PASSED in 38.7s Stats over 50 runs: max = 38.7s, min = 3.6s, avg = 16.3s, dev = 8.0s //tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test PASSED in 39.9s Stats over 50 runs: max = 39.9s, min = 3.6s, avg = 11.2s, dev = 8.0s //tensorflow/compiler/tests:unary_ops_test_cpu PASSED in 14.9s Stats over 50 runs: max = 14.9s, min = 3.8s, avg = 6.4s, dev = 2.8s //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test PASSED in 29.1s Stats over 50 runs: max = 29.1s, min = 3.6s, avg = 7.5s, dev = 4.9s //tensorflow/compiler/xla/tests:conv_depthwise_test_cpu PASSED in 9.6s Stats over 50 runs: max = 9.6s, min = 6.7s, avg = 7.7s, dev = 0.8s //tensorflow/compiler/xla/tests:convolution_test_1d_no_vmodule_cpu PASSED in 12.8s Stats over 50 runs: max = 12.8s, min = 7.8s, avg = 10.2s, dev = 1.3s //tensorflow/compiler/xla/tests:convolution_test_cpu PASSED in 15.4s Stats over 50 runs: max = 15.4s, min = 8.3s, avg = 11.5s, dev = 1.7s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 15.0s Stats over 50 runs: max = 15.0s, min = 4.7s, avg = 9.0s, dev = 2.7s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_grad_test_cpu PASSED in 11.0s Stats over 50 runs: max = 11.0s, min = 4.0s, avg = 6.0s, dev = 2.0s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_sparse_mat_mul_grad_test_cpu PASSED in 10.9s Stats over 50 runs: max = 10.9s, min = 3.9s, avg = 7.2s, dev = 2.1s //tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu PASSED in 33.9s Stats over 50 runs: max = 33.9s, min = 7.6s, avg = 17.4s, dev = 6.5s //tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu PASSED in 15.5s Stats over 50 runs: max = 15.5s, min = 3.8s, avg = 6.7s, dev = 2.5s //tensorflow/python/kernel_tests/math_ops:cwise_ops_unary_test_cpu PASSED in 15.7s Stats over 50 runs: max = 15.7s, min = 3.6s, avg = 7.1s, dev = 2.4s Executed 3893 out of 3893 tests: 3777 tests pass and 116 fail locally. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.