==================== Test output for //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test (shard 2 of 8): Running tests under Python 3.9.16: /usr/local/bin/python3 [ RUN ] SnapshotFtTest.testLargeMultiSourceSnapshotRecoversAndCompletes_test_mode_eager_tfapiversion_2 2023-03-10 20:16:45.682326: I tensorflow/core/data/service/dispatcher_impl.cc:228] Attempting to restore dispatcher state from journal in /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e912086183a5_f0_2u/tmpy12ubhgo/tf_data_dispatcher_journal 2023-03-10 20:16:45.682403: I tensorflow/core/data/service/dispatcher_impl.cc:235] No journal found. Starting dispatcher from new state. 2023-03-10 20:16:45.682680: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data DispatchServer running at 0.0.0.0:44723 2023-03-10 20:16:45.697574: I tensorflow/core/data/service/worker_impl.cc:181] Worker registered with dispatcher running at localhost:44723 2023-03-10 20:16:45.697831: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:42175 2023-03-10 20:16:45.700125: I tensorflow/core/data/service/worker_impl.cc:181] Worker registered with dispatcher running at localhost:44723 2023-03-10 20:16:45.700338: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:45663 2023-03-10 20:16:45.702452: I tensorflow/core/data/service/worker_impl.cc:181] Worker registered with dispatcher running at localhost:44723 2023-03-10 20:16:45.702678: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:33495 2023-03-10 20:16:45.704562: I tensorflow/core/data/service/worker_impl.cc:181] Worker registered with dispatcher running at localhost:44723 2023-03-10 20:16:45.704789: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:44825 2023-03-10 20:16:45.706673: I tensorflow/core/data/service/worker_impl.cc:181] Worker registered with dispatcher running at localhost:44723 2023-03-10 20:16:45.706886: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:45167 2023-03-10 20:16:45.760027: I tensorflow/core/data/service/server_lib.cc:94] Shut down DispatchServer server running at port 44723 2023-03-10 20:16:45.760774: I tensorflow/core/data/service/dispatcher_impl.cc:228] Attempting to restore dispatcher state from journal in /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e912086183a5_f0_2u/tmpy12ubhgo/tf_data_dispatcher_journal 2023-03-10 20:16:45.798915: W tensorflow/core/data/service/worker_impl.cc:540] Failed to send heartbeat to dispatcher: UNAVAILABLE: Failed to perform worker heartbeat: Socket closed 2023-03-10 20:16:45.800874: W tensorflow/core/data/service/worker_impl.cc:540] Failed to send heartbeat to dispatcher: UNAVAILABLE: Failed to perform worker heartbeat: Socket closed 2023-03-10 20:16:45.803168: W tensorflow/core/data/service/worker_impl.cc:540] Failed to send heartbeat to dispatcher: UNAVAILABLE: Failed to perform worker heartbeat: Socket closed 2023-03-10 20:16:45.805225: W tensorflow/core/data/service/worker_impl.cc:540] Failed to send heartbeat to dispatcher: UNAVAILABLE: Failed to perform worker heartbeat: Socket closed 2023-03-10 20:16:45.816873: W tensorflow/core/data/service/worker_impl.cc:540] Failed to send heartbeat to dispatcher: UNAVAILABLE: Failed to perform worker heartbeat: Socket closed 2023-03-10 20:16:45.837562: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data DispatchServer running at 0.0.0.0:44723 2023-03-10 20:16:45.962834: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:84] Writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e912086183a5_f0_2u/tmp8afuutyo/tmpue712daw/snapshot_ft_test, stream: 1, compression: SNAPPY } 2023-03-10 20:16:45.963262: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:136] Writing distributed tf.data snapshot stream 1, chunk 0. 2023-03-10 20:16:45.963300: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:84] Writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e912086183a5_f0_2u/tmp8afuutyo/tmpue712daw/snapshot_ft_test, stream: 0, compression: SNAPPY } 2023-03-10 20:16:45.963722: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:136] Writing distributed tf.data snapshot stream 0, chunk 0. 2023-03-10 20:16:45.973567: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:84] Writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e912086183a5_f0_2u/tmp8afuutyo/tmpue712daw/snapshot_ft_test, stream: 2, compression: SNAPPY } 2023-03-10 20:16:45.974022: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:136] Writing distributed tf.data snapshot stream 2, chunk 0. 2023-03-10 20:16:45.990943: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:84] Writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e912086183a5_f0_2u/tmp8afuutyo/tmpue712daw/snapshot_ft_test, stream: 3, compression: SNAPPY } 2023-03-10 20:16:45.991051: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:84] Writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e912086183a5_f0_2u/tmp8afuutyo/tmpue712daw/snapshot_ft_test, stream: 4, compression: SNAPPY } 2023-03-10 20:16:45.991444: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:136] Writing distributed tf.data snapshot stream 3, chunk 0. 2023-03-10 20:16:46.006238: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:136] Writing distributed tf.data snapshot stream 4, chunk 0. 2023-03-10 20:17:06.739769: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:100] Finished writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e912086183a5_f0_2u/tmp8afuutyo/tmpue712daw/snapshot_ft_test, stream: 4, compression: SNAPPY } 2023-03-10 20:17:06.740029: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:100] Finished writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e912086183a5_f0_2u/tmp8afuutyo/tmpue712daw/snapshot_ft_test, stream: 1, compression: SNAPPY } 2023-03-10 20:17:06.740194: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:100] Finished writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e912086183a5_f0_2u/tmp8afuutyo/tmpue712daw/snapshot_ft_test, stream: 2, compression: SNAPPY } 2023-03-10 20:17:06.741064: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:100] Finished writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e912086183a5_f0_2u/tmp8afuutyo/tmpue712daw/snapshot_ft_test, stream: 0, compression: SNAPPY } 2023-03-10 20:17:06.741356: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:100] Finished writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e912086183a5_f0_2u/tmp8afuutyo/tmpue712daw/snapshot_ft_test, stream: 3, compression: SNAPPY } 2023-03-10 20:17:06.840887: I tensorflow/core/data/service/snapshot/snapshot_manager.cc:251] Finished writing tf.data distributed snapshot at /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e912086183a5_f0_2u/tmp8afuutyo/tmpue712daw/snapshot_ft_test Fatal Python error: Bus error Current thread 0x0000ffff89db7380 (most recent call first): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/service/server_lib.py", line 446 in _stop File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/service/server_lib.py", line 449 in __del__ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/test_base.py", line 251 in __del__ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343 in execute_test_method File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360 in decorated File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/parameterized.py", line 314 in bound_param_test File "/usr/lib/python3.9/unittest/case.py", line 550 in _callTestMethod File "/usr/lib/python3.9/unittest/case.py", line 592 in run File "/usr/lib/python3.9/unittest/case.py", line 651 in __call__ File "/usr/lib/python3.9/unittest/suite.py", line 122 in run File "/usr/lib/python3.9/unittest/suite.py", line 84 in __call__ File "/usr/lib/python3.9/unittest/suite.py", line 122 in run File "/usr/lib/python3.9/unittest/suite.py", line 84 in __call__ File "/usr/lib/python3.9/unittest/runner.py", line 184 in run File "/usr/lib/python3.9/unittest/main.py", line 271 in runTests File "/usr/lib/python3.9/unittest/main.py", line 101 in __init__ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2537 in _run_and_get_tests_result File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2568 in run_tests File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2156 in _run_in_app File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2049 in main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51 in g_main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/app.py", line 258 in _run_main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/app.py", line 312 in run File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60 in main_wrapper File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489 in benchmarks_main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62 in main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56 in main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.py", line 232 in *** Received signal 7 *** *** BEGIN MANGLED STACK TRACE *** Fatal Python error: Segmentation fault Current thread 0x0000ffff89db7380 (most recent call first): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/service/server_lib.py", line 446 in _stop File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/service/server_lib.py", line 449 in __del__ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/test_base.py", line 251 in __del__ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343 in execute_test_method File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360 in decorated File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/parameterized.py", line 314 in bound_param_test File "/usr/lib/python3.9/unittest/case.py", line 550 in _callTestMethod File "/usr/lib/python3.9/unittest/case.py", line 592 in run File "/usr/lib/python3.9/unittest/case.py", line 651 in __call__ File "/usr/lib/python3.9/unittest/suite.py", line 122 in run File "/usr/lib/python3.9/unittest/suite.py", line 84 in __call__ File "/usr/lib/python3.9/unittest/suite.py", line 122 in run File "/usr/lib/python3.9/unittest/suite.py", line 84 in __call__ File "/usr/lib/python3.9/unittest/runner.py", line 184 in run File "/usr/lib/python3.9/unittest/main.py", line 271 in runTests File "/usr/lib/python3.9/unittest/main.py", line 101 in __init__ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2537 in _run_and_get_tests_result File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2568 in run_tests File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2156 in _run_in_app File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2049 in main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51 in g_main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/app.py", line 258 in _run_main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/app.py", line 312 in run File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60 in main_wrapper File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489 in benchmarks_main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62 in main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56 in main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.py", line 232 in *** Received signal 11 *** *** BEGIN MANGLED STACK TRACE *** ================================================================================ ==================== Test output for //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test (shard 2 of 8): Running tests under Python 3.9.16: /usr/local/bin/python3 [ RUN ] SnapshotFtTest.testLargeMultiSourceSnapshotRecoversAndCompletes_test_mode_eager_tfapiversion_2 2023-03-10 20:17:13.232784: I tensorflow/core/data/service/dispatcher_impl.cc:228] Attempting to restore dispatcher state from journal in /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e9120861831tq1x5_m/tmph4bi9aog/tf_data_dispatcher_journal 2023-03-10 20:17:13.232881: I tensorflow/core/data/service/dispatcher_impl.cc:235] No journal found. Starting dispatcher from new state. 2023-03-10 20:17:13.245863: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data DispatchServer running at 0.0.0.0:44751 2023-03-10 20:17:13.251636: I tensorflow/core/data/service/worker_impl.cc:181] Worker registered with dispatcher running at localhost:44751 2023-03-10 20:17:13.251861: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:38181 2023-03-10 20:17:13.254103: I tensorflow/core/data/service/worker_impl.cc:181] Worker registered with dispatcher running at localhost:44751 2023-03-10 20:17:13.254309: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:33807 2023-03-10 20:17:13.256079: I tensorflow/core/data/service/worker_impl.cc:181] Worker registered with dispatcher running at localhost:44751 2023-03-10 20:17:13.256305: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:42705 2023-03-10 20:17:13.258478: I tensorflow/core/data/service/worker_impl.cc:181] Worker registered with dispatcher running at localhost:44751 2023-03-10 20:17:13.258698: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:37029 2023-03-10 20:17:13.260220: I tensorflow/core/data/service/worker_impl.cc:181] Worker registered with dispatcher running at localhost:44751 2023-03-10 20:17:13.260404: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:42993 2023-03-10 20:17:13.554045: I tensorflow/core/data/service/server_lib.cc:94] Shut down DispatchServer server running at port 44751 2023-03-10 20:17:13.554922: I tensorflow/core/data/service/dispatcher_impl.cc:228] Attempting to restore dispatcher state from journal in /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e9120861831tq1x5_m/tmph4bi9aog/tf_data_dispatcher_journal 2023-03-10 20:17:13.660592: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:84] Writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e9120861831tq1x5_m/tmpacwb8h7d/tmpcip4gq2k/snapshot_ft_test, stream: 0, compression: SNAPPY } 2023-03-10 20:17:13.660983: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:136] Writing distributed tf.data snapshot stream 0, chunk 0. 2023-03-10 20:17:13.663146: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:84] Writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e9120861831tq1x5_m/tmpacwb8h7d/tmpcip4gq2k/snapshot_ft_test, stream: 1, compression: SNAPPY } 2023-03-10 20:17:13.663549: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:136] Writing distributed tf.data snapshot stream 1, chunk 0. 2023-03-10 20:17:13.667141: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:84] Writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e9120861831tq1x5_m/tmpacwb8h7d/tmpcip4gq2k/snapshot_ft_test, stream: 2, compression: SNAPPY } 2023-03-10 20:17:13.667530: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:136] Writing distributed tf.data snapshot stream 2, chunk 0. 2023-03-10 20:17:13.668605: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data DispatchServer running at 0.0.0.0:44751 2023-03-10 20:17:13.673991: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:84] Writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e9120861831tq1x5_m/tmpacwb8h7d/tmpcip4gq2k/snapshot_ft_test, stream: 4, compression: SNAPPY } 2023-03-10 20:17:13.674445: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:136] Writing distributed tf.data snapshot stream 4, chunk 0. 2023-03-10 20:17:13.683658: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:84] Writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e9120861831tq1x5_m/tmpacwb8h7d/tmpcip4gq2k/snapshot_ft_test, stream: 3, compression: SNAPPY } 2023-03-10 20:17:13.684079: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:136] Writing distributed tf.data snapshot stream 3, chunk 0. 2023-03-10 20:17:13.761308: W tensorflow/core/data/service/worker_impl.cc:540] Failed to send heartbeat to dispatcher: UNAVAILABLE: Failed to perform worker heartbeat: Socket closed 2023-03-10 20:17:35.076984: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:100] Finished writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e9120861831tq1x5_m/tmpacwb8h7d/tmpcip4gq2k/snapshot_ft_test, stream: 3, compression: SNAPPY } 2023-03-10 20:17:35.077343: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:100] Finished writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e9120861831tq1x5_m/tmpacwb8h7d/tmpcip4gq2k/snapshot_ft_test, stream: 4, compression: SNAPPY } 2023-03-10 20:17:35.082786: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:100] Finished writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e9120861831tq1x5_m/tmpacwb8h7d/tmpcip4gq2k/snapshot_ft_test, stream: 2, compression: SNAPPY } 2023-03-10 20:17:35.096806: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:100] Finished writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e9120861831tq1x5_m/tmpacwb8h7d/tmpcip4gq2k/snapshot_ft_test, stream: 1, compression: SNAPPY } 2023-03-10 20:17:35.097202: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:100] Finished writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e9120861831tq1x5_m/tmpacwb8h7d/tmpcip4gq2k/snapshot_ft_test, stream: 0, compression: SNAPPY } 2023-03-10 20:17:35.176864: I tensorflow/core/data/service/snapshot/snapshot_manager.cc:251] Finished writing tf.data distributed snapshot at /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e9120861831tq1x5_m/tmpacwb8h7d/tmpcip4gq2k/snapshot_ft_test Fatal Python error: Segmentation fault Current thread 0x0000ffffb79a7380 (most recent call first): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/service/server_lib.py", line 446 in _stop File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/service/server_lib.py", line 449 in __del__ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/test_base.py", line 251 in __del__ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343 in execute_test_method File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360 in decorated File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/parameterized.py", line 314 in bound_param_test File "/usr/lib/python3.9/unittest/case.py", line 550 in _callTestMethod File "/usr/lib/python3.9/unittest/case.py", line 592 in run File "/usr/lib/python3.9/unittest/case.py", line 651 in __call__ File "/usr/lib/python3.9/unittest/suite.py", line 122 in run File "/usr/lib/python3.9/unittest/suite.py", line 84 in __call__ File "/usr/lib/python3.9/unittest/suite.py", line 122 in run File "/usr/lib/python3.9/unittest/suite.py", line 84 in __call__ File "/usr/lib/python3.9/unittest/runner.py", line 184 in run File "/usr/lib/python3.9/unittest/main.py", line 271 in runTests File "/usr/lib/python3.9/unittest/main.py", line 101 in __init__ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2537 in _run_and_get_tests_result File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2568 in run_tests File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2156 in _run_in_app File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2049 in main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51 in g_main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/app.py", line 258 in _run_main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/app.py", line 312 in run File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60 in main_wrapper File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489 in benchmarks_main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62 in main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56 in main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.py", line 232 in *** Received signal 11 *** *** BEGIN MANGLED STACK TRACE *** /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/../../../_solib_local/_U_S_Stensorflow_Clibtensorflow_Uframework_Uimport_Ulib___Utensorflow/libtensorflow_framework.so.2(+0x157d89c)[0xffffb584d89c] linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffffb79607a0] /lib/aarch64-linux-gnu/libc.so.6(gsignal+0xe0)[0xffffb77e3d78] linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffffb79607a0] /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/../../../_solib_local/_U_S_Stensorflow_Clibtensorflow_Uframework_Uimport_Ulib___Utensorflow/libtensorflow_framework.so.2(_ZNSt10_HashtableINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4pairIKS5_N4absl12lts_202206234CordEESaISB_ENSt8__detail10_Select1stESt8equal_toIS5_ESt4hashIS5_ENSD_18_Mod_range_hashingENSD_20_Default_ranged_hashENSD_20_Prime_rehash_policyENSD_17_Hashtable_traitsILb1ELb0ELb1EEEED1Ev+0x1c)[0xffffb49fac2c] /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/../../../_solib_local/_U_S_Stensorflow_Clibtensorflow_Uframework_Uimport_Ulib___Utensorflow/libtensorflow_framework.so.2(_ZN3tsl6Status5StateD1Ev+0x24)[0xffffb49fad14] /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/../../../_solib_local/_U_S_Stensorflow_Clibtensorflow_Uframework_Uimport_Ulib___Utensorflow/libtensorflow_framework.so.2(_ZN3tsl6Status12SlowCopyFromEPKNS0_5StateE+0x44)[0xffffb49febf4] ================================================================================ ==================== Test output for //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test (shard 2 of 8): Running tests under Python 3.9.16: /usr/local/bin/python3 [ RUN ] SnapshotFtTest.testLargeMultiSourceSnapshotRecoversAndCompletes_test_mode_eager_tfapiversion_2 2023-03-10 20:17:43.316901: I tensorflow/core/data/service/dispatcher_impl.cc:228] Attempting to restore dispatcher state from journal in /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e91208618347qpihvu/tmpb55qtpnv/tf_data_dispatcher_journal 2023-03-10 20:17:43.316990: I tensorflow/core/data/service/dispatcher_impl.cc:235] No journal found. Starting dispatcher from new state. 2023-03-10 20:17:43.317262: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data DispatchServer running at 0.0.0.0:33135 2023-03-10 20:17:43.320056: I tensorflow/core/data/service/worker_impl.cc:181] Worker registered with dispatcher running at localhost:33135 2023-03-10 20:17:43.320284: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:36935 2023-03-10 20:17:43.322646: I tensorflow/core/data/service/worker_impl.cc:181] Worker registered with dispatcher running at localhost:33135 2023-03-10 20:17:43.322847: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:41303 2023-03-10 20:17:43.324648: I tensorflow/core/data/service/worker_impl.cc:181] Worker registered with dispatcher running at localhost:33135 2023-03-10 20:17:43.324855: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:42817 2023-03-10 20:17:43.327257: I tensorflow/core/data/service/worker_impl.cc:181] Worker registered with dispatcher running at localhost:33135 2023-03-10 20:17:43.327491: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:44991 2023-03-10 20:17:43.330860: I tensorflow/core/data/service/worker_impl.cc:181] Worker registered with dispatcher running at localhost:33135 2023-03-10 20:17:43.331097: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:35907 2023-03-10 20:17:43.427000: W tensorflow/core/data/service/worker_impl.cc:540] Failed to send heartbeat to dispatcher: UNAVAILABLE: Failed to perform worker heartbeat: Socket closed 2023-03-10 20:17:43.427163: I tensorflow/core/data/service/server_lib.cc:94] Shut down DispatchServer server running at port 33135 2023-03-10 20:17:43.427798: I tensorflow/core/data/service/dispatcher_impl.cc:228] Attempting to restore dispatcher state from journal in /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e91208618347qpihvu/tmpb55qtpnv/tf_data_dispatcher_journal 2023-03-10 20:17:43.434382: W tensorflow/core/data/service/worker_impl.cc:540] Failed to send heartbeat to dispatcher: UNAVAILABLE: Failed to perform worker heartbeat: Socket closed 2023-03-10 20:17:43.440524: W tensorflow/core/data/service/worker_impl.cc:540] Failed to send heartbeat to dispatcher: UNAVAILABLE: Failed to perform worker heartbeat: Socket closed 2023-03-10 20:17:43.456793: W tensorflow/core/data/service/worker_impl.cc:540] Failed to send heartbeat to dispatcher: UNAVAILABLE: Failed to perform worker heartbeat: Socket closed 2023-03-10 20:17:43.482607: W tensorflow/core/data/service/worker_impl.cc:540] Failed to send heartbeat to dispatcher: UNAVAILABLE: Failed to perform worker heartbeat: Socket closed 2023-03-10 20:17:43.588119: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data DispatchServer running at 0.0.0.0:33135 2023-03-10 20:17:43.636095: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:84] Writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e91208618347qpihvu/tmpmhjjx5zq/tmpq3t3xjtr/snapshot_ft_test, stream: 1, compression: SNAPPY } 2023-03-10 20:17:43.636560: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:136] Writing distributed tf.data snapshot stream 1, chunk 0. 2023-03-10 20:17:43.644906: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:84] Writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e91208618347qpihvu/tmpmhjjx5zq/tmpq3t3xjtr/snapshot_ft_test, stream: 2, compression: SNAPPY } 2023-03-10 20:17:43.645388: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:136] Writing distributed tf.data snapshot stream 2, chunk 0. 2023-03-10 20:17:43.646196: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:84] Writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e91208618347qpihvu/tmpmhjjx5zq/tmpq3t3xjtr/snapshot_ft_test, stream: 0, compression: SNAPPY } 2023-03-10 20:17:43.646741: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:136] Writing distributed tf.data snapshot stream 0, chunk 0. 2023-03-10 20:17:43.659312: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:84] Writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e91208618347qpihvu/tmpmhjjx5zq/tmpq3t3xjtr/snapshot_ft_test, stream: 3, compression: SNAPPY } 2023-03-10 20:17:43.659757: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:136] Writing distributed tf.data snapshot stream 3, chunk 0. 2023-03-10 20:17:43.666020: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:84] Writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e91208618347qpihvu/tmpmhjjx5zq/tmpq3t3xjtr/snapshot_ft_test, stream: 4, compression: SNAPPY } 2023-03-10 20:17:43.716503: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:136] Writing distributed tf.data snapshot stream 4, chunk 0. 2023-03-10 20:18:20.960357: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:100] Finished writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e91208618347qpihvu/tmpmhjjx5zq/tmpq3t3xjtr/snapshot_ft_test, stream: 3, compression: SNAPPY } 2023-03-10 20:18:20.961733: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:100] Finished writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e91208618347qpihvu/tmpmhjjx5zq/tmpq3t3xjtr/snapshot_ft_test, stream: 2, compression: SNAPPY } 2023-03-10 20:18:20.962992: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:100] Finished writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e91208618347qpihvu/tmpmhjjx5zq/tmpq3t3xjtr/snapshot_ft_test, stream: 0, compression: SNAPPY } 2023-03-10 20:18:20.964915: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:100] Finished writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e91208618347qpihvu/tmpmhjjx5zq/tmpq3t3xjtr/snapshot_ft_test, stream: 4, compression: SNAPPY } 2023-03-10 20:18:20.964978: I tensorflow/core/data/service/snapshot/snapshot_stream_writer.cc:100] Finished writing distributed tf.data snapshot stream: SnapshotWriterParams { base_path: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e91208618347qpihvu/tmpmhjjx5zq/tmpq3t3xjtr/snapshot_ft_test, stream: 1, compression: SNAPPY } 2023-03-10 20:18:21.036886: I tensorflow/core/data/service/snapshot/snapshot_manager.cc:251] Finished writing tf.data distributed snapshot at /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/efbab71aa75682c5a19f95e91208618347qpihvu/tmpmhjjx5zq/tmpq3t3xjtr/snapshot_ft_test Fatal Python error: Segmentation fault Current thread 0x0000ffffa2d47380 (most recent call first): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/service/server_lib.py", line 446 in _stop File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/service/server_lib.py", line 449 in __del__ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/test_base.py", line 251 in __del__ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343 in execute_test_method File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360 in decorated File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/parameterized.py", line 314 in bound_param_test File "/usr/lib/python3.9/unittest/case.py", line 550 in _callTestMethod File "/usr/lib/python3.9/unittest/case.py", line 592 in run File "/usr/lib/python3.9/unittest/case.py", line 651 in __call__ File "/usr/lib/python3.9/unittest/suite.py", line 122 in run File "/usr/lib/python3.9/unittest/suite.py", line 84 in __call__ File "/usr/lib/python3.9/unittest/suite.py", line 122 in run File "/usr/lib/python3.9/unittest/suite.py", line 84 in __call__ File "/usr/lib/python3.9/unittest/runner.py", line 184 in run File "/usr/lib/python3.9/unittest/main.py", line 271 in runTests File "/usr/lib/python3.9/unittest/main.py", line 101 in __init__ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2537 in _run_and_get_tests_result File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2568 in run_tests File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2156 in _run_in_app File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2049 in main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51 in g_main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/app.py", line 258 in _run_main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/app.py", line 312 in run File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60 in main_wrapper File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489 in benchmarks_main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62 in main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56 in main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.py", line 232 in *** Received signal 11 *** *** BEGIN MANGLED STACK TRACE *** /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/../../../_solib_local/_U_S_Stensorflow_Clibtensorflow_Uframework_Uimport_Ulib___Utensorflow/libtensorflow_framework.so.2(+0x157d89c)[0xffffa0bed89c] linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffffa2d007a0] /lib/aarch64-linux-gnu/libc.so.6(gsignal+0xe0)[0xffffa2b83d78] linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffffa2d007a0] [0xfffcb801af30] *** END MANGLED STACK TRACE *** *** Begin stack trace *** tsl::CurrentStackTrace[abi:cxx11]() __kernel_rt_sigreturn gsignal __kernel_rt_sigreturn *** End stack trace *** ================================================================================ ==================== Test output for //tensorflow/python/distribute/failure_handling:failure_handler_test (shard 1 of 8): Running tests under Python 3.9.16: /usr/local/bin/python3 [ RUN ] PreemptionCheckpointTest.test_error_propagation INFO:tensorflow:Using local port 24645 I0310 20:17:31.357536 281473343124352 test_util.py:3797] Using local port 24645 INFO:tensorflow:Using local port 22258 I0310 20:17:31.358527 281473343124352 test_util.py:3797] Using local port 22258 INFO:tensorflow:Using local port 21912 I0310 20:17:31.358931 281473343124352 test_util.py:3797] Using local port 21912 INFO:tensorflow:Using local port 18462 I0310 20:17:31.359321 281473343124352 test_util.py:3797] Using local port 18462 INFO:tensorflow:Cluster starting. I0310 20:17:35.827769 281473343124352 failure_handler_test.py:381] Cluster starting. [worker-0]: I0310 20:17:35.921134 281473253340032 multi_process_runner.py:840] Subprocess with PID 1012531 (worker, 0) is now being started. [worker-0]: I0310 20:17:35.921532 281473253340032 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24645", "localhost:22258", "localhost:21912", "localhost:18462"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: I0310 20:17:36.073899 281473253340032 multi_process_runner.py:840] Subprocess with PID 1012799 (worker, 2) is now being started. [worker-1]: I0310 20:17:36.077090 281473253340032 multi_process_runner.py:840] Subprocess with PID 1012624 (worker, 1) is now being started. [worker-3]: I0310 20:17:36.091261 281473253340032 multi_process_runner.py:840] Subprocess with PID 1012857 (worker, 3) is now being started. [worker-0]: 2023-03-10 20:17:36.093323: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:24645 [worker-2]: I0310 20:17:36.074301 281473253340032 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24645", "localhost:22258", "localhost:21912", "localhost:18462"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-1]: I0310 20:17:36.077487 281473253340032 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24645", "localhost:22258", "localhost:21912", "localhost:18462"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-3]: I0310 20:17:36.091658 281473253340032 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24645", "localhost:22258", "localhost:21912", "localhost:18462"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2023-03-10 20:17:36.241743: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 17338930727307309859 [worker-0]: 2023-03-10 20:17:36.256169: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-1]: 2023-03-10 20:17:36.267846: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:22258 [worker-0]: 2023-03-10 20:17:36.271147: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 9978725297754914616 [worker-1]: 2023-03-10 20:17:36.271407: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-2]: 2023-03-10 20:17:36.307231: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:21912 [worker-0]: 2023-03-10 20:17:36.310649: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 6049657261739275163 [worker-2]: 2023-03-10 20:17:36.311262: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-3]: 2023-03-10 20:17:36.446728: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:18462 [worker-0]: 2023-03-10 20:17:36.450161: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 5946601342424295973 [worker-3]: 2023-03-10 20:17:36.450775: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0310 20:17:36.453959 281473253340032 collective_all_reduce_strategy.py:532] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0310 20:17:36.486895 281473253340032 collective_all_reduce_strategy.py:532] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0310 20:17:36.460448 281473253340032 collective_all_reduce_strategy.py:532] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0310 20:17:36.467010 281473253340032 collective_all_reduce_strategy.py:532] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0310 20:17:36.572907 281473253340032 collective_all_reduce_strategy.py:575] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24645', 'localhost:22258', 'localhost:21912', 'localhost:18462']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0310 20:17:36.573315 281473253340032 collective_all_reduce_strategy.py:577] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24645', 'localhost:22258', 'localhost:21912', 'localhost:18462']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0310 20:17:36.588979 281473253340032 collective_all_reduce_strategy.py:575] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24645', 'localhost:22258', 'localhost:21912', 'localhost:18462']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0310 20:17:36.589282 281473253340032 collective_all_reduce_strategy.py:577] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24645', 'localhost:22258', 'localhost:21912', 'localhost:18462']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0310 20:17:36.613161 281473253340032 collective_all_reduce_strategy.py:575] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24645', 'localhost:22258', 'localhost:21912', 'localhost:18462']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0310 20:17:36.613584 281473253340032 collective_all_reduce_strategy.py:577] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24645', 'localhost:22258', 'localhost:21912', 'localhost:18462']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0310 20:17:36.633421 281473253340032 collective_all_reduce_strategy.py:575] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24645', 'localhost:22258', 'localhost:21912', 'localhost:18462']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0310 20:17:36.633723 281473253340032 collective_all_reduce_strategy.py:577] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24645', 'localhost:22258', 'localhost:21912', 'localhost:18462']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0310 20:17:36.765901 281473253340032 failure_handling.py:633] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0310 20:17:36.767055 281473253340032 failure_handling.py:633] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I0310 20:17:36.770916 281473253340032 failure_handling.py:673] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0310 20:17:36.771199 281473253340032 failure_handling.py:537] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-2]: I0310 20:17:36.767998 281473253340032 failure_handling.py:673] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0310 20:17:36.768208 281473253340032 failure_handling.py:537] PreemptionCheckpointHandler initialized or restored. [worker-1]: I0310 20:17:36.788333 281473253340032 failure_handling.py:633] Start watcher for peer's signal. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0310 20:17:36.795877 281473253340032 failure_handling.py:633] Start watcher for peer's signal. [worker-1]: I0310 20:17:36.788926 281473253340032 failure_handling.py:673] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0310 20:17:36.789132 281473253340032 failure_handling.py:537] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: W0310 20:17:36.771471 281473253340032 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-2]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0310 20:17:36.768474 281473253340032 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: W0310 20:17:36.789393 281473253340032 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-1]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-0]: INFO:tensorflow:Start training at 0 [worker-2]: I0310 20:17:36.768628 281473253340032 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Start training at 0 [worker-0]: I0310 20:17:36.771625 281473253340032 failure_handler_test.py:197] Start training at 0 [worker-1]: I0310 20:17:36.789545 281473253340032 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I0310 20:17:36.815378 281473253340032 failure_handling.py:673] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0310 20:17:36.815679 281473253340032 failure_handling.py:537] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0310 20:17:36.815943 281473253340032 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0310 20:17:36.816101 281473253340032 failure_handler_test.py:197] Start training at 0 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:37.000866 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:37.061348 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Error reported to Coordinator: in user code: [worker-2]: [worker-0]: 2023-03-10 20:17:37.176479: E tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:961] /job:worker/replica:0/task:2 has been set to ERROR in coordination service: RESOURCE_EXHAUSTED: in user code: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: [worker-1]: 2023-03-10 20:17:37.186304: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-0]: ResourceExhaustedError: Running out of resources [worker-1]: [worker-0]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-1]: ResourceExhaustedError: Running out of resources [worker-0]: 2023-03-10 20:17:37.179171: E tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:737] Coordination agent is in ERROR: RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-1]: [worker-0]: [worker-1]: The error could be from a previous operation. Restart your program to reset. [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: Additional GRPC error information from remote target /job:worker/replica:0/task:0: [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: :{"created":"@1678479457.179719477","description":"Error received from peer ipv6:[::1]:24645","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Collective ops is aborted by: Error reported from /job:worker/task:2: in user code:\n\n File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn *\n return call_for_each_replica(strategy, fn.python_function, args, kwargs)\n File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step *\n raise errors_impl.ResourceExhaustedError(\n\n ResourceExhaustedError: Running out of resources\n\nThe error could be from a previous operation. Restart your program to reset.","grpc_status":8} [type.googleapis.com/tensorflow.DerivedStatus=''] [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: 2023-03-10 20:17:37.186422: I tensorflow/core/common_runtime/executor.cc:1210] [/job:worker/replica:0/task:1/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): RESOURCE_EXHAUSTED: Collective ops is aborted by: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-1]: [worker-0]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: ResourceExhaustedError: Running out of resources [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: 2023-03-10 20:17:37.179205: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-0]: [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: ResourceExhaustedError: Running out of resources [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: The error could be from a previous operation. Restart your program to reset. [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-1]: Additional GRPC error information from remote target /job:worker/replica:0/task:0: [worker-0]: [worker-1]: :{"created":"@1678479457.179719477","description":"Error received from peer ipv6:[::1]:24645","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Collective ops is aborted by: Error reported from /job:worker/task:2: in user code:\n\n File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn *\n return call_for_each_replica(strategy, fn.python_function, args, kwargs)\n File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step *\n raise errors_impl.ResourceExhaustedError(\n\n ResourceExhaustedError: Running out of resources\n\nThe error could be from a previous operation. Restart your program to reset.","grpc_status":8} [worker-0]: ResourceExhaustedError: Running out of resources [worker-1]: The error could be from a previous operation. Restart your program to reset. [worker-0]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-1]: [[{{node CollectiveReduceV2}}]] [worker-0]: 2023-03-10 20:17:37.179260: I tensorflow/core/common_runtime/executor.cc:1210] [/job:worker/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): RESOURCE_EXHAUSTED: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-1]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-0]: [worker-1]: [type.googleapis.com/tensorflow.DerivedStatus=''] [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-0]: [worker-0]: ResourceExhaustedError: Running out of resources [worker-0]: [worker-0]: The error could be from a previous operation. Restart your program to reset. [worker-0]: [[{{node CollectiveReduceV2}}]] [worker-0]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-0]: [type.googleapis.com/tensorflow.DerivedStatus=''] [worker-0]: 2023-03-10 20:17:37.179312: E tensorflow/core/distributed_runtime/worker.cc:431] Bad status from CompleteGroupDistributed: RESOURCE_EXHAUSTED: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-0]: [worker-0]: ResourceExhaustedError: Running out of resources [worker-0]: [worker-0]: The error could be from a previous operation. Restart your program to reset. [type.googleapis.com/tensorflow.DerivedStatus=''] [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-3]: 2023-03-10 20:17:37.196331: E tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:737] Coordination agent is in ERROR: RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-2]: [worker-3]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: Traceback (most recent call last): [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/training/coordinator.py", line 293, in stop_on_exception [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: yield [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 387, in run [worker-3]: [worker-2]: self.main_result = self.main_fn(*self.main_args, **self.main_kwargs) [worker-3]: ResourceExhaustedError: Running out of resources [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/autograph/impl/api.py", line 692, in wrapper [worker-3]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-2]: raise e.ag_error_metadata.to_exception(e) [worker-3]: 2023-03-10 20:17:37.196402: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-2]: tensorflow.python.framework.errors_impl.ResourceExhaustedError: in user code: [worker-3]: [worker-1]: 2023-03-10 20:17:37.199100: E tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:737] Coordination agent is in ERROR: RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-2]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-2]: ResourceExhaustedError: Running out of resources [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: [worker-3]: [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-2]: I0310 20:17:37.172838 281447243837920 coordinator.py:213] Error reported to Coordinator: in user code: [worker-3]: ResourceExhaustedError: Running out of resources [worker-1]: [worker-2]: [worker-3]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-1]: ResourceExhaustedError: Running out of resources [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: Traceback (most recent call last): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/training/coordinator.py", line 293, in stop_on_exception [worker-2]: yield [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 387, in run [worker-2]: self.main_result = self.main_fn(*self.main_args, **self.main_kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/autograph/impl/api.py", line 692, in wrapper [worker-2]: raise e.ag_error_metadata.to_exception(e) [worker-2]: tensorflow.python.framework.errors_impl.ResourceExhaustedError: in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [worker-2]: INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [worker-2]: I0310 20:17:37.175428 281473253340032 failure_handling.py:913] Propagating error to cluster: ResourceExhaustedError(): in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [worker-2]: 2023-03-10 20:17:37.175707: E tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:737] Coordination agent is in ERROR: RESOURCE_EXHAUSTED: in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-2]: 2023-03-10 20:17:37.175756: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-2]: 2023-03-10 20:17:37.175793: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:433] Reporting error to coordination service: RESOURCE_EXHAUSTED: in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:37.342061 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: 2023-03-10 20:17:37.406290: I tensorflow/core/common_runtime/executor.cc:1210] [/job:worker/replica:0/task:3/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): RESOURCE_EXHAUSTED: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-3]: [worker-3]: ResourceExhaustedError: Running out of resources [worker-3]: [worker-3]: The error could be from a previous operation. Restart your program to reset. [worker-3]: [[{{node CollectiveReduceV2}}]] [worker-3]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-3]: [type.googleapis.com/tensorflow.DerivedStatus=''] [worker-1]: INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-1]: [worker-1]: Detected at node 'CollectiveReduceV2' defined at (most recent call last): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 552, in [worker-1]: test_util.main() [worker-0]: INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-0]: [worker-0]: Detected at node 'CollectiveReduceV2' defined at (most recent call last): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 552, in [worker-0]: test_util.main() [worker-1]: File "", line 1, in [worker-0]: File "", line 1, in [worker-1]: File "/usr/lib/python3.9/multiprocessing/forkserver.py", line 274, in main [worker-0]: File "/usr/lib/python3.9/multiprocessing/forkserver.py", line 274, in main [worker-1]: code = _serve_one(child_r, fds, [worker-0]: code = _serve_one(child_r, fds, [worker-0]: File "/usr/lib/python3.9/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-0]: File "/usr/lib/python3.9/multiprocessing/spawn.py", line 129, in _main [worker-0]: return self._bootstrap(parent_sentinel) [worker-0]: File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap [worker-0]: self.run() [worker-0]: File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 361, in assert_raise_error [worker-0]: self.worker_fn( [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-0]: preemption_handler.run(distributed_train_step, epoch, step) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-0]: strategy.run(train_step) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: Node: 'CollectiveReduceV2' [worker-0]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-0]: [worker-0]: ResourceExhaustedError: Running out of resources [worker-0]: [worker-0]: The error could be from a previous operation. Restart your program to reset. [worker-0]: [[{{node CollectiveReduceV2}}]] [worker-0]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-0]: [Op:__inference_train_step_41] [worker-0]: I0310 20:17:37.987210 281473253340032 failure_handling.py:913] Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-0]: [worker-0]: Detected at node 'CollectiveReduceV2' defined at (most recent call last): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 552, in [worker-0]: test_util.main() [worker-0]: File "", line 1, in [worker-0]: File "/usr/lib/python3.9/multiprocessing/forkserver.py", line 274, in main [worker-0]: code = _serve_one(child_r, fds, [worker-0]: File "/usr/lib/python3.9/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-0]: File "/usr/lib/python3.9/multiprocessing/spawn.py", line 129, in _main [worker-0]: return self._bootstrap(parent_sentinel) [worker-1]: File "/usr/lib/python3.9/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-1]: File "/usr/lib/python3.9/multiprocessing/spawn.py", line 129, in _main [worker-1]: return self._bootstrap(parent_sentinel) [worker-1]: File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap [worker-0]: self.run() [worker-1]: self.run() [worker-1]: File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-0]: File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 361, in assert_raise_error [worker-0]: self._target(*self._args, **self._kwargs) [worker-1]: self.worker_fn( [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 361, in assert_raise_error [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-0]: self.worker_fn( [worker-1]: preemption_handler.run(distributed_train_step, epoch, step) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-1]: strategy.run(train_step) [worker-0]: preemption_handler.run(distributed_train_step, epoch, step) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: strategy.run(train_step) [worker-1]: Node: 'CollectiveReduceV2' [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-1]: Collective ops is aborted by: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: [worker-0]: Node: 'CollectiveReduceV2' [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: ResourceExhaustedError: Running out of resources [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-0]: [worker-0]: ResourceExhaustedError: Running out of resources [worker-0]: [worker-0]: The error could be from a previous operation. Restart your program to reset. [worker-0]: [[{{node CollectiveReduceV2}}]] [worker-0]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-0]: [Op:__inference_train_step_41] [worker-0]: INFO:tensorflow:Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-0]: I0310 20:17:37.987587 281473253340032 failure_handling.py:917] Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-1]: [worker-1]: The error could be from a previous operation. Restart your program to reset. [worker-1]: Additional GRPC error information from remote target /job:worker/replica:0/task:0: [worker-1]: :{"created":"@1678479457.179719477","description":"Error received from peer ipv6:[::1]:24645","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Collective ops is aborted by: Error reported from /job:worker/task:2: in user code:\n\n File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn *\n return call_for_each_replica(strategy, fn.python_function, args, kwargs)\n File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step *\n raise errors_impl.ResourceExhaustedError(\n\n ResourceExhaustedError: Running out of resources\n\nThe error could be from a previous operation. Restart your program to reset.","grpc_status":8} [worker-1]: The error could be from a previous operation. Restart your program to reset. [worker-1]: [[{{node CollectiveReduceV2}}]] [worker-1]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-1]: [Op:__inference_train_step_39] [worker-1]: I0310 20:17:37.984823 281473253340032 failure_handling.py:913] Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-1]: [worker-1]: Detected at node 'CollectiveReduceV2' defined at (most recent call last): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 552, in [worker-1]: test_util.main() [worker-1]: File "", line 1, in [worker-1]: File "/usr/lib/python3.9/multiprocessing/forkserver.py", line 274, in main [worker-1]: code = _serve_one(child_r, fds, [worker-1]: File "/usr/lib/python3.9/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-1]: File "/usr/lib/python3.9/multiprocessing/spawn.py", line 129, in _main [worker-1]: return self._bootstrap(parent_sentinel) [worker-1]: File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap [worker-1]: self.run() [worker-1]: File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 361, in assert_raise_error [worker-1]: self.worker_fn( [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-1]: preemption_handler.run(distributed_train_step, epoch, step) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-1]: strategy.run(train_step) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: Node: 'CollectiveReduceV2' [worker-1]: Collective ops is aborted by: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-1]: [worker-1]: ResourceExhaustedError: Running out of resources [worker-1]: [worker-1]: The error could be from a previous operation. Restart your program to reset. [worker-1]: Additional GRPC error information from remote target /job:worker/replica:0/task:0: [worker-1]: :{"created":"@1678479457.179719477","description":"Error received from peer ipv6:[::1]:24645","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Collective ops is aborted by: Error reported from /job:worker/task:2: in user code:\n\n File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn *\n return call_for_each_replica(strategy, fn.python_function, args, kwargs)\n File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step *\n raise errors_impl.ResourceExhaustedError(\n\n ResourceExhaustedError: Running out of resources\n\nThe error could be from a previous operation. Restart your program to reset.","grpc_status":8} [worker-1]: The error could be from a previous operation. Restart your program to reset. [worker-1]: [[{{node CollectiveReduceV2}}]] [worker-1]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-1]: [Op:__inference_train_step_39] [worker-1]: INFO:tensorflow:Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-1]: I0310 20:17:37.985358 281473253340032 failure_handling.py:917] Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-3]: Exception in thread PeerTerminationWatcher-3: [worker-3]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.9/threading.py", line 980, in _bootstrap_inner [worker-3]: self.run() [worker-3]: File "/usr/lib/python3.9/threading.py", line 917, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 1222, in _watch_step_to_save_key [worker-3]: step_value = context.context().get_config_key_value(_INITIAL_RUN_COUNT_KEY) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 788, in get_config_key_value [worker-3]: pywrap_tfe.TFE_GetConfigKeyValue(self._context_handle, key, [worker-3]: tensorflow.python.framework.errors_impl.UnavailableError: Socket closed [worker-3]: Additional GRPC error information from remote target /job:worker/replica:0/task:0: [worker-3]: :{"created":"@1678479458.102702957","description":"Error received from peer ipv6:[::1]:24645","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed","grpc_status":14} [worker-3]: INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-3]: [worker-3]: Detected at node 'CollectiveReduceV2' defined at (most recent call last): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 552, in [worker-3]: test_util.main() [worker-3]: File "", line 1, in [worker-3]: File "/usr/lib/python3.9/multiprocessing/forkserver.py", line 274, in main [worker-3]: code = _serve_one(child_r, fds, [worker-3]: File "/usr/lib/python3.9/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-3]: File "/usr/lib/python3.9/multiprocessing/spawn.py", line 129, in _main [worker-3]: return self._bootstrap(parent_sentinel) [worker-3]: File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap [worker-3]: self.run() [worker-3]: File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 361, in assert_raise_error [worker-3]: self.worker_fn( [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-3]: preemption_handler.run(distributed_train_step, epoch, step) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-3]: strategy.run(train_step) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: Node: 'CollectiveReduceV2' [worker-3]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-3]: [worker-3]: ResourceExhaustedError: Running out of resources [worker-3]: [worker-3]: The error could be from a previous operation. Restart your program to reset. [worker-3]: [[{{node CollectiveReduceV2}}]] [worker-3]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-3]: [Op:__inference_train_step_39] [worker-3]: I0310 20:17:38.319418 281473253340032 failure_handling.py:913] Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-3]: [worker-3]: Detected at node 'CollectiveReduceV2' defined at (most recent call last): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 552, in [worker-3]: test_util.main() [worker-3]: File "", line 1, in [worker-3]: File "/usr/lib/python3.9/multiprocessing/forkserver.py", line 274, in main [worker-3]: code = _serve_one(child_r, fds, [worker-3]: File "/usr/lib/python3.9/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-3]: File "/usr/lib/python3.9/multiprocessing/spawn.py", line 129, in _main [worker-3]: return self._bootstrap(parent_sentinel) [worker-3]: File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap [worker-3]: self.run() [worker-3]: File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 361, in assert_raise_error [worker-3]: self.worker_fn( [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-3]: preemption_handler.run(distributed_train_step, epoch, step) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-3]: strategy.run(train_step) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: Node: 'CollectiveReduceV2' [worker-3]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-3]: [worker-3]: ResourceExhaustedError: Running out of resources [worker-3]: [worker-3]: The error could be from a previous operation. Restart your program to reset. [worker-3]: [[{{node CollectiveReduceV2}}]] [worker-3]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-3]: [Op:__inference_train_step_39] [worker-3]: INFO:tensorflow:Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-3]: I0310 20:17:38.319809 281473253340032 failure_handling.py:917] Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. I0310 20:17:38.877780 281473343124352 multi_process_runner.py:646] worker-0 exit code: 0 I0310 20:17:38.878074 281473343124352 multi_process_runner.py:646] worker-1 exit code: 0 I0310 20:17:38.878211 281473343124352 multi_process_runner.py:646] worker-2 exit code: 0 I0310 20:17:38.878335 281473343124352 multi_process_runner.py:646] worker-3 exit code: 0 I0310 20:17:38.881444 281473343124352 multi_process_runner.py:662] Joining log reading threads. I0310 20:17:38.881671 281473343124352 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_error_propagation): 7.69s I0310 20:17:39.029998 281473343124352 test_util.py:2467] time(__main__.PreemptionCheckpointTest.test_error_propagation): 7.69s [ OK ] PreemptionCheckpointTest.test_error_propagation [ RUN ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice INFO:tensorflow:Start watcher for local signal. I0310 20:17:39.136838 281473343124352 failure_handling.py:673] Start watcher for local signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. I0310 20:17:39.137256 281473343124352 failure_handling.py:537] PreemptionCheckpointHandler initialized or restored. WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. W0310 20:17:39.137539 281473343124352 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. INFO:tensorflow:Start training at 0 I0310 20:17:39.137716 281473343124352 failure_handler_test.py:197] Start training at 0 WARNING:tensorflow:5 out of the last 5 calls to .distributed_train_step..train_step at 0xfffef426be50> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0310 20:17:39.430021 281473343124352 polymorphic_function.py:157] 5 out of the last 5 calls to .distributed_train_step..train_step at 0xfffef426be50> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. WARNING:tensorflow:6 out of the last 6 calls to .distributed_train_step..train_step at 0xfffef426be50> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0310 20:17:39.442905 281473343124352 polymorphic_function.py:157] 6 out of the last 6 calls to .distributed_train_step..train_step at 0xfffef426be50> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. INFO:tensorflow:epoch 0 finished I0310 20:17:39.562578 281473343124352 failure_handler_test.py:195] epoch 0 finished INFO:tensorflow:epoch 1 finished I0310 20:17:39.756732 281473343124352 failure_handler_test.py:195] epoch 1 finished INFO:tensorflow:epoch 2 finished I0310 20:17:40.003300 281473343124352 failure_handler_test.py:195] epoch 2 finished INFO:tensorflow:epoch 3 finished I0310 20:17:40.284046 281473343124352 failure_handler_test.py:195] epoch 3 finished INFO:tensorflow:epoch 4 finished I0310 20:17:40.529564 281473343124352 failure_handler_test.py:195] epoch 4 finished INFO:tensorflow:epoch 5 finished I0310 20:17:40.755106 281473343124352 failure_handler_test.py:195] epoch 5 finished INFO:tensorflow:epoch 6 finished I0310 20:17:40.934360 281473343124352 failure_handler_test.py:195] epoch 6 finished INFO:tensorflow:sending sigterm I0310 20:17:41.052290 281472404877792 failure_handler_test.py:461] sending sigterm --- Logging error --- Traceback (most recent call last): File "/usr/lib/python3.9/logging/__init__.py", line 1083, in emit msg = self.format(record) File "/usr/lib/python3.9/logging/__init__.py", line 927, in format return fmt.format(record) File "/usr/lib/python3.9/logging/__init__.py", line 663, in format record.message = record.getMessage() File "/usr/lib/python3.9/logging/__init__.py", line 367, in getMessage msg = msg % self.args TypeError: not all arguments converted during string formatting Call stack: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 552, in test_util.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/test_util.py", line 137, in main multi_process_runner.test_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1455, in test_main multi_process_lib.test_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 167, in test_main test.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/test.py", line 25, in main _test.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.9/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.9/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.9/unittest/runner.py", line 184, in run test(result) File "/usr/lib/python3.9/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.9/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.9/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.9/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.9/unittest/case.py", line 651, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.9/unittest/case.py", line 592, in run self._callTestMethod(testMethod) File "/usr/lib/python3.9/unittest/case.py", line 550, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 471, in test_grace_period_continue_training self.worker_fn(checkpoint_dir, cluster_spec, strategy_option, input_arg, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn preemption_handler.run(distributed_train_step, epoch, step) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 888, in run return self._run_for_multi_worker_mirrored( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 904, in _run_for_multi_worker_mirrored result = distributed_train_function(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step strategy.run(train_step) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/one_device_strategy.py", line 187, in run return super(OneDeviceStrategy, self).run(fn, args, kwargs, options) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/distribute_lib.py", line 1289, in run return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/distribute_lib.py", line 2866, in call_for_each_replica return self._call_for_each_replica(fn, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/one_device_strategy.py", line 399, in _call_for_each_replica return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 839, in __call__ result = self._call(*args, **kwds) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 911, in _call return self._concrete_variable_creation_fn._call_flat( # pylint: disable=protected-access File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/polymorphic_function/monomorphic_function.py", line 1344, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/polymorphic_function/atomic_function.py", line 286, in call outputs = execute.execute( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/execute.py", line 52, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 1213, in _sigterm_handler_fn self._maybe_set_received_own_sigterm() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 700, in _maybe_set_received_own_sigterm logging.info('Received termination notice.', File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/platform/tf_logging.py", line 198, in info get_logger().info(msg, *args, **kwargs) Message: 'Received termination notice.' Arguments: ('single_worker',) --- Logging error --- Traceback (most recent call last): File "/usr/lib/python3.9/logging/__init__.py", line 1083, in emit msg = self.format(record) File "/usr/lib/python3.9/logging/__init__.py", line 927, in format return fmt.format(record) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/logging/__init__.py", line 1025, in format return prefix + super(PythonFormatter, self).format(record) File "/usr/lib/python3.9/logging/__init__.py", line 663, in format record.message = record.getMessage() File "/usr/lib/python3.9/logging/__init__.py", line 367, in getMessage msg = msg % self.args TypeError: not all arguments converted during string formatting Call stack: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 552, in test_util.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/test_util.py", line 137, in main multi_process_runner.test_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1455, in test_main multi_process_lib.test_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 167, in test_main test.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/test.py", line 25, in main _test.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.9/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.9/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.9/unittest/runner.py", line 184, in run test(result) File "/usr/lib/python3.9/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.9/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.9/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.9/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.9/unittest/case.py", line 651, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.9/unittest/case.py", line 592, in run self._callTestMethod(testMethod) File "/usr/lib/python3.9/unittest/case.py", line 550, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 471, in test_grace_period_continue_training self.worker_fn(checkpoint_dir, cluster_spec, strategy_option, input_arg, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn preemption_handler.run(distributed_train_step, epoch, step) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 888, in run return self._run_for_multi_worker_mirrored( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 904, in _run_for_multi_worker_mirrored result = distributed_train_function(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step strategy.run(train_step) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/one_device_strategy.py", line 187, in run return super(OneDeviceStrategy, self).run(fn, args, kwargs, options) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/distribute_lib.py", line 1289, in run return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/distribute_lib.py", line 2866, in call_for_each_replica return self._call_for_each_replica(fn, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/one_device_strategy.py", line 399, in _call_for_each_replica return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 839, in __call__ result = self._call(*args, **kwds) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 911, in _call return self._concrete_variable_creation_fn._call_flat( # pylint: disable=protected-access File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/polymorphic_function/monomorphic_function.py", line 1344, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/polymorphic_function/atomic_function.py", line 286, in call outputs = execute.execute( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/execute.py", line 52, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 1213, in _sigterm_handler_fn self._maybe_set_received_own_sigterm() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 700, in _maybe_set_received_own_sigterm logging.info('Received termination notice.', File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/platform/tf_logging.py", line 198, in info get_logger().info(msg, *args, **kwargs) File "/usr/lib/python3.9/logging/__init__.py", line 1446, in info self._log(INFO, msg, args, **kwargs) File "/usr/lib/python3.9/logging/__init__.py", line 1589, in _log self.handle(record) File "/usr/lib/python3.9/logging/__init__.py", line 1599, in handle self.callHandlers(record) File "/usr/lib/python3.9/logging/__init__.py", line 1661, in callHandlers hdlr.handle(record) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/logging/__init__.py", line 988, in handle return self._current_handler.handle(record) File "/usr/lib/python3.9/logging/__init__.py", line 952, in handle self.emit(record) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/logging/__init__.py", line 925, in emit super(PythonHandler, self).emit(record) Message: 'Received termination notice.' Arguments: ('single_worker',) INFO:tensorflow:Termination caught in main thread on preempted worker I0310 20:17:41.199028 281473343124352 failure_handling.py:1152] Termination caught in main thread on preempted worker INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. I0310 20:17:41.211417 281473343124352 failure_handling.py:1057] PreemptionCheckpointHandler: Starting saving a checkpoint. INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90g6csyzkj/tmpq8466iv0/fh_ckpt I0310 20:17:41.278370 281473343124352 failure_handling.py:1071] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90g6csyzkj/tmpq8466iv0/fh_ckpt INFO:tensorflow:Continue training for the grace period. I0310 20:17:41.278719 281473343124352 failure_handling.py:1127] Continue training for the grace period. INFO:tensorflow:epoch 7 finished I0310 20:17:41.361675 281473343124352 failure_handler_test.py:195] epoch 7 finished INFO:tensorflow:Training finished. I0310 20:17:41.362378 281473343124352 failure_handler_test.py:245] Training finished. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice): 2.33s I0310 20:17:41.363275 281473343124352 test_util.py:2467] time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice): 2.33s [ OK ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 17862 I0310 20:17:41.366095 281473343124352 test_util.py:3797] Using local port 17862 INFO:tensorflow:Using local port 20636 I0310 20:17:41.366626 281473343124352 test_util.py:3797] Using local port 20636 INFO:tensorflow:Using local port 16684 I0310 20:17:41.367027 281473343124352 test_util.py:3797] Using local port 16684 INFO:tensorflow:Using local port 19578 I0310 20:17:41.367416 281473343124352 test_util.py:3797] Using local port 19578 INFO:tensorflow:Cluster starting. I0310 20:17:41.396779 281473343124352 failure_handler_test.py:291] Cluster starting. [worker-1]: I0310 20:17:41.444214 281473253340032 multi_process_runner.py:840] Subprocess with PID 1032713 (worker, 1) is now being started. [worker-0]: I0310 20:17:41.445185 281473253340032 multi_process_runner.py:840] Subprocess with PID 1032704 (worker, 0) is now being started. [worker-1]: I0310 20:17:41.444664 281473253340032 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:17862", "localhost:20636", "localhost:16684", "localhost:19578"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: I0310 20:17:41.445648 281473253340032 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:17862", "localhost:20636", "localhost:16684", "localhost:19578"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: 2023-03-10 20:17:41.482714: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:20636 [worker-0]: 2023-03-10 20:17:41.486057: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:17862 [worker-0]: 2023-03-10 20:17:41.489193: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 10094810869400812371 [worker-1]: 2023-03-10 20:17:41.495745: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: 2023-03-10 20:17:41.498840: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 2997204040464968098 [worker-0]: 2023-03-10 20:17:41.499084: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-2]: I0310 20:17:41.501407 281473253340032 multi_process_runner.py:840] Subprocess with PID 1032738 (worker, 2) is now being started. [worker-2]: I0310 20:17:41.501833 281473253340032 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:17862", "localhost:20636", "localhost:16684", "localhost:19578"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0310 20:17:41.503221 281473253340032 multi_process_runner.py:840] Subprocess with PID 1032749 (worker, 3) is now being started. [worker-3]: I0310 20:17:41.503640 281473253340032 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:17862", "localhost:20636", "localhost:16684", "localhost:19578"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-3]: 2023-03-10 20:17:41.544704: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:19578 [worker-0]: 2023-03-10 20:17:41.552846: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 1571923941453723129 [worker-3]: 2023-03-10 20:17:41.553092: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-2]: 2023-03-10 20:17:41.562085: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:16684 [worker-0]: 2023-03-10 20:17:41.570330: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 6390533050504407589 [worker-2]: 2023-03-10 20:17:41.570555: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0310 20:17:41.572609 281473253340032 collective_all_reduce_strategy.py:532] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0310 20:17:41.577105 281473253340032 collective_all_reduce_strategy.py:532] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: I0310 20:17:41.577114 281473253340032 collective_all_reduce_strategy.py:532] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0310 20:17:41.580305 281473253340032 collective_all_reduce_strategy.py:532] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0310 20:17:41.631111 281473253340032 collective_all_reduce_strategy.py:575] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17862', 'localhost:20636', 'localhost:16684', 'localhost:19578']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0310 20:17:41.631875 281473253340032 collective_all_reduce_strategy.py:575] Check health not enabled. [worker-2]: I0310 20:17:41.631402 281473253340032 collective_all_reduce_strategy.py:577] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17862', 'localhost:20636', 'localhost:16684', 'localhost:19578']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17862', 'localhost:20636', 'localhost:16684', 'localhost:19578']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0310 20:17:41.632174 281473253340032 collective_all_reduce_strategy.py:577] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17862', 'localhost:20636', 'localhost:16684', 'localhost:19578']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0310 20:17:41.631446 281473253340032 collective_all_reduce_strategy.py:575] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17862', 'localhost:20636', 'localhost:16684', 'localhost:19578']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0310 20:17:41.631800 281473253340032 collective_all_reduce_strategy.py:577] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17862', 'localhost:20636', 'localhost:16684', 'localhost:19578']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0310 20:17:41.631112 281473253340032 collective_all_reduce_strategy.py:575] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17862', 'localhost:20636', 'localhost:16684', 'localhost:19578']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0310 20:17:41.631426 281473253340032 collective_all_reduce_strategy.py:577] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17862', 'localhost:20636', 'localhost:16684', 'localhost:19578']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0310 20:17:41.717947 281473253340032 failure_handling.py:633] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0310 20:17:41.727709 281473253340032 failure_handling.py:633] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0310 20:17:41.738128 281473253340032 failure_handling.py:633] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0310 20:17:41.744421 281473253340032 failure_handling.py:673] Start watcher for local signal. [worker-0]: I0310 20:17:41.744504 281473253340032 failure_handling.py:673] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0310 20:17:41.744801 281473253340032 failure_handling.py:537] PreemptionCheckpointHandler initialized or restored. [worker-0]: I0310 20:17:41.744843 281473253340032 failure_handling.py:537] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0310 20:17:41.745129 281473253340032 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I0310 20:17:41.747816 281473253340032 failure_handling.py:633] Start watcher for peer's signal. [worker-0]: W0310 20:17:41.745158 281473253340032 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-0]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: I0310 20:17:41.746509 281473253340032 failure_handling.py:673] Start watcher for local signal. [worker-3]: I0310 20:17:41.749218 281473253340032 failure_handling.py:673] Start watcher for local signal. [worker-1]: INFO:tensorflow:Start training at 0 [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0310 20:17:41.749405 281473253340032 failure_handling.py:537] PreemptionCheckpointHandler initialized or restored. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: I0310 20:17:41.746794 281473253340032 failure_handling.py:537] PreemptionCheckpointHandler initialized or restored. [worker-1]: I0310 20:17:41.745298 281473253340032 failure_handler_test.py:197] Start training at 0 [worker-0]: INFO:tensorflow:Start training at 0 [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: I0310 20:17:41.745319 281473253340032 failure_handler_test.py:197] Start training at 0 [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: Instructions for updating: [worker-2]: W0310 20:17:41.747055 281473253340032 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Instructions for updating: [worker-3]: W0310 20:17:41.749652 281473253340032 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: Instructions for updating: [worker-2]: INFO:tensorflow:Start training at 0 [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: I0310 20:17:41.747213 281473253340032 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0310 20:17:41.749808 281473253340032 failure_handler_test.py:197] Start training at 0 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:42.024913 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:42.028597 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:42.041424 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:42.040829 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:42.226093 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:42.237182 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:42.241836 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:42.242151 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:42.331288 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:42.362536 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:42.380222 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:42.481296 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:42.602067 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:42.600888 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:42.620807 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:42.624238 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:42.765918 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:42.781088 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:42.810448 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:42.818752 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff95630af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff95634af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff95630af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0310 20:17:42.901240 281473253340032 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff95630af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0310 20:17:42.901452 281473253340032 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff95634af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff95634af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0310 20:17:42.901697 281473253340032 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff95634af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0310 20:17:42.901632 281473253340032 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff95630af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:42.910904 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:42.911069 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:42.911624 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:42.912046 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff956430d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0310 20:17:43.000659 281473253340032 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff956430d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff956420d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0310 20:17:43.002055 281473253340032 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff956420d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff956410d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff956410d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0310 20:17:43.011097 281473253340032 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff956410d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0310 20:17:43.004029 281473253340032 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff956410d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:43.176679 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:43.177486 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:43.220081 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:43.284657 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:43.343711 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:43.345111 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:43.357521 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:43.358916 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:43.440677 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:43.451773 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:43.480420 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:43.516070 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:43.569334 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:43.569463 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:43.569927 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:43.574817 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:43.625521 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:43.635763 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:43.648809 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:43.645380 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:43.698424 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:43.698468 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:43.699168 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:43.698842 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:43.747975 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:43.747786 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:43.747786 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:43.748542 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:43.843936 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:43.838451 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:43.850427 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:43.863135 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:43.924986 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:43.925292 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:43.932126 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:43.946630 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0310 20:17:44.023780 281473253340032 failure_handler_test.py:195] epoch 0 finished [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0310 20:17:44.026702 281473253340032 failure_handler_test.py:195] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:44.034187 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0310 20:17:44.036842 281473253340032 failure_handler_test.py:195] epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0310 20:17:44.041587 281473253340032 failure_handler_test.py:195] epoch 0 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:44.045379 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:44.052217 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:44.036063 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:44.265985 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:44.275690 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:44.288599 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:44.310470 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:44.359603 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:44.359601 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:44.360347 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:44.381432 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:44.439571 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:44.440168 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:44.439674 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:44.440267 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending sigterm I0310 20:17:44.457738 281473343124352 failure_handler_test.py:296] sending sigterm INFO:tensorflow:sigterm sent I0310 20:17:44.458121 281473343124352 failure_handler_test.py:300] sigterm sent [worker-2]: INFO:tensorflow:Member 2 has received termination notice. [worker-2]: I0310 20:17:44.458700 281473253340032 failure_handling.py:709] Member 2 has received termination notice. [worker-2]: INFO:tensorflow:Termination caught in main thread on preempted worker [worker-2]: I0310 20:17:44.482230 281473253340032 failure_handling.py:1152] Termination caught in main thread on preempted worker [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-2]: I0310 20:17:44.485353 281447495365088 failure_handling.py:1235] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-3]: I0310 20:17:44.485651 281447629582816 failure_handling.py:1235] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-0]: I0310 20:17:44.486301 281446706967008 failure_handling.py:1235] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:RUN_TO_CHECKPOINT set to 20 [worker-2]: I0310 20:17:44.486719 281473253340032 failure_handling.py:1161] RUN_TO_CHECKPOINT set to 20 [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 0 received [worker-1]: I0310 20:17:44.487699 281447503819232 failure_handling.py:1235] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-2]: I0310 20:17:44.488130 281473253340032 failure_handling.py:1170] Sigterm acknowledgement from replica 0 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 1 received [worker-2]: I0310 20:17:44.488798 281473253340032 failure_handling.py:1170] Sigterm acknowledgement from replica 1 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 2 received [worker-2]: I0310 20:17:44.489331 281473253340032 failure_handling.py:1170] Sigterm acknowledgement from replica 2 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 3 received [worker-2]: I0310 20:17:44.489844 281473253340032 failure_handling.py:1170] Sigterm acknowledgement from replica 3 received [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:44.492621 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:44.497902 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:44.507454 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:44.518395 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: I0310 20:17:44.606694 281473253340032 failure_handling.py:1057] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: I0310 20:17:44.596536 281473253340032 failure_handling.py:1057] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: I0310 20:17:44.621008 281473253340032 failure_handling.py:1057] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: I0310 20:17:44.626602 281473253340032 failure_handling.py:1057] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90g6csyzkj/tmplm4tz6bq/workertemp_3/fh_ckpt [worker-3]: I0310 20:17:44.642237 281473253340032 failure_handling.py:1071] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90g6csyzkj/tmplm4tz6bq/workertemp_3/fh_ckpt [worker-3]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-3]: I0310 20:17:44.644339 281473253340032 failure_handling.py:766] Shut down watcher for peer's termination signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: I0310 20:17:44.644544 281473253340032 failure_handling.py:1121] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90g6csyzkj/tmplm4tz6bq/fh_ckpt [worker-0]: I0310 20:17:44.649532 281473253340032 failure_handling.py:1071] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90g6csyzkj/tmplm4tz6bq/fh_ckpt [worker-0]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-0]: I0310 20:17:44.651540 281473253340032 failure_handling.py:766] Shut down watcher for peer's termination signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: I0310 20:17:44.651792 281473253340032 failure_handling.py:1121] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-1]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90g6csyzkj/tmplm4tz6bq/workertemp_1/fh_ckpt [worker-1]: I0310 20:17:44.664274 281473253340032 failure_handling.py:1071] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90g6csyzkj/tmplm4tz6bq/workertemp_1/fh_ckpt [worker-2]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90g6csyzkj/tmplm4tz6bq/workertemp_2/fh_ckpt [worker-2]: I0310 20:17:44.672512 281473253340032 failure_handling.py:1071] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90g6csyzkj/tmplm4tz6bq/workertemp_2/fh_ckpt [worker-1]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-1]: I0310 20:17:44.685518 281473253340032 failure_handling.py:766] Shut down watcher for peer's termination signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-1]: I0310 20:17:44.685801 281473253340032 failure_handling.py:1121] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-2]: I0310 20:17:44.707293 281473253340032 failure_handling.py:766] Shut down watcher for peer's termination signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: I0310 20:17:44.707558 281473253340032 failure_handling.py:1121] PreemptionCheckpointHandler: checkpoint saved. Exiting. INFO:tensorflow:restarting workers I0310 20:17:46.462258 281473343124352 failure_handler_test.py:303] restarting workers [worker-0]: I0310 20:17:46.505574 281473253340032 multi_process_runner.py:840] Subprocess with PID 1054756 (worker, 0) is now being started. INFO:tensorflow:workers restarted I0310 20:17:46.512664 281473343124352 failure_handler_test.py:307] workers restarted [worker-0]: I0310 20:17:46.506041 281473253340032 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:17862", "localhost:20636", "localhost:16684", "localhost:19578"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-0]: 2023-03-10 20:17:46.619329: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:17862 [worker-2]: I0310 20:17:46.643666 281473253340032 multi_process_runner.py:840] Subprocess with PID 1054771 (worker, 2) is now being started. [worker-2]: I0310 20:17:46.644082 281473253340032 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:17862", "localhost:20636", "localhost:16684", "localhost:19578"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0310 20:17:46.654630 281473253340032 multi_process_runner.py:840] Subprocess with PID 1054776 (worker, 3) is now being started. [worker-3]: I0310 20:17:46.655034 281473253340032 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:17862", "localhost:20636", "localhost:16684", "localhost:19578"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-1]: I0310 20:17:46.674896 281473253340032 multi_process_runner.py:840] Subprocess with PID 1054766 (worker, 1) is now being started. [worker-1]: I0310 20:17:46.675321 281473253340032 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:17862", "localhost:20636", "localhost:16684", "localhost:19578"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: 2023-03-10 20:17:46.690995: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 12149386584630937861 [worker-0]: 2023-03-10 20:17:46.693455: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-3]: 2023-03-10 20:17:46.712295: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:19578 [worker-2]: 2023-03-10 20:17:46.723703: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:16684 [worker-0]: 2023-03-10 20:17:46.729384: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 9685427350536364169 [worker-3]: 2023-03-10 20:17:46.729607: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: 2023-03-10 20:17:46.732265: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 4263940046782370018 [worker-2]: 2023-03-10 20:17:46.732483: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-1]: 2023-03-10 20:17:46.766788: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:20636 [worker-0]: 2023-03-10 20:17:46.771602: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 3143914069487384754 [worker-1]: 2023-03-10 20:17:46.771820: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0310 20:17:46.773602 281473253340032 collective_all_reduce_strategy.py:532] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0310 20:17:46.773752 281473253340032 collective_all_reduce_strategy.py:532] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0310 20:17:46.777118 281473253340032 collective_all_reduce_strategy.py:532] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0310 20:17:46.781090 281473253340032 collective_all_reduce_strategy.py:532] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0310 20:17:46.831300 281473253340032 collective_all_reduce_strategy.py:575] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17862', 'localhost:20636', 'localhost:16684', 'localhost:19578']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0310 20:17:46.831625 281473253340032 collective_all_reduce_strategy.py:577] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17862', 'localhost:20636', 'localhost:16684', 'localhost:19578']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0310 20:17:46.835711 281473253340032 collective_all_reduce_strategy.py:575] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17862', 'localhost:20636', 'localhost:16684', 'localhost:19578']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0310 20:17:46.836010 281473253340032 collective_all_reduce_strategy.py:577] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17862', 'localhost:20636', 'localhost:16684', 'localhost:19578']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0310 20:17:46.833447 281473253340032 collective_all_reduce_strategy.py:575] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17862', 'localhost:20636', 'localhost:16684', 'localhost:19578']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0310 20:17:46.833765 281473253340032 collective_all_reduce_strategy.py:577] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17862', 'localhost:20636', 'localhost:16684', 'localhost:19578']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0310 20:17:46.857292 281473253340032 collective_all_reduce_strategy.py:575] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17862', 'localhost:20636', 'localhost:16684', 'localhost:19578']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0310 20:17:46.857581 281473253340032 collective_all_reduce_strategy.py:577] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:17862', 'localhost:20636', 'localhost:16684', 'localhost:19578']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0310 20:17:46.913120 281473253340032 failure_handling.py:633] Start watcher for peer's signal. [worker-1]: I0310 20:17:46.913116 281473253340032 failure_handling.py:633] Start watcher for peer's signal. [worker-0]: I0310 20:17:46.913121 281473253340032 failure_handling.py:633] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I0310 20:17:46.913815 281473253340032 failure_handling.py:673] Start watcher for local signal. [worker-0]: I0310 20:17:46.914233 281473253340032 failure_handling.py:673] Start watcher for local signal. [worker-3]: I0310 20:17:46.913115 281473253340032 failure_handling.py:633] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0310 20:17:46.914006 281473253340032 failure_handling.py:537] PreemptionCheckpointHandler initialized or restored. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: I0310 20:17:46.914462 281473253340032 failure_handling.py:537] PreemptionCheckpointHandler initialized or restored. [worker-2]: Instructions for updating: [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0310 20:17:46.914258 281473253340032 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: W0310 20:17:46.914725 281473253340032 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-0]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-2]: INFO:tensorflow:Start training at 20 [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: I0310 20:17:46.914414 281473253340032 failure_handler_test.py:197] Start training at 20 [worker-0]: INFO:tensorflow:Start training at 20 [worker-1]: I0310 20:17:46.926931 281473253340032 failure_handling.py:673] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: INFO:tensorflow:training restarted [worker-0]: I0310 20:17:46.914881 281473253340032 failure_handler_test.py:197] Start training at 20 [worker-1]: I0310 20:17:46.927216 281473253340032 failure_handling.py:537] PreemptionCheckpointHandler initialized or restored. [worker-0]: INFO:tensorflow:training restarted [worker-2]: I0310 20:17:46.917999 281473253340032 failure_handler_test.py:207] training restarted [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: I0310 20:17:46.918847 281473253340032 failure_handler_test.py:207] training restarted [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0310 20:17:46.927478 281473253340032 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 20 [worker-1]: I0310 20:17:46.927637 281473253340032 failure_handler_test.py:197] Start training at 20 [worker-1]: INFO:tensorflow:training restarted [worker-1]: I0310 20:17:46.963136 281473253340032 failure_handler_test.py:207] training restarted [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I0310 20:17:47.006315 281473253340032 failure_handling.py:673] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0310 20:17:47.006628 281473253340032 failure_handling.py:537] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0310 20:17:47.006909 281473253340032 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 20 [worker-3]: I0310 20:17:47.007069 281473253340032 failure_handler_test.py:197] Start training at 20 [worker-3]: INFO:tensorflow:training restarted [worker-3]: I0310 20:17:47.008893 281473253340032 failure_handler_test.py:207] training restarted [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:47.109858 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:47.168172 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:47.213108 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:47.224893 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:47.361372 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:47.377091 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:47.380300 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:47.380347 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:47.460478 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:47.479321 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:47.468746 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:47.508769 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:47.574932 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:47.579450 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:47.604258 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:47.589483 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:47.658795 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:47.665941 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:47.688085 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:47.689649 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff95642af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0310 20:17:47.734933 281473253340032 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff95642af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff95646af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff95641af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0310 20:17:47.747045 281473253340032 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff95646af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0310 20:17:47.747877 281473253340032 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff95641af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff95643af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0310 20:17:47.754653 281473253340032 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff95643af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:47.841415 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:47.859081 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:47.887509 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:47.908812 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff955d10d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0310 20:17:47.977504 281473253340032 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff955d10d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff955d00d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff955d20d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0310 20:17:47.987460 281473253340032 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff955d00d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0310 20:17:47.987971 281473253340032 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff955d20d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff955d00d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0310 20:17:47.996588 281473253340032 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff955d00d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:47.989196 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:48.005963 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:48.006297 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:48.016717 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:48.086415 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:48.089883 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:48.086466 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:48.100481 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:48.150969 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:48.150968 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:48.154578 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:48.151590 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:48.207116 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:48.207247 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:48.218923 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:48.219588 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-3]: I0310 20:17:48.276926 281473253340032 failure_handler_test.py:195] epoch 1 finished [worker-0]: I0310 20:17:48.277111 281473253340032 failure_handler_test.py:195] epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0310 20:17:48.277940 281473253340032 failure_handler_test.py:195] epoch 1 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:48.288034 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:48.278243 281473253340032 failure_handler_test.py:195] epoch 1 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:48.287544 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:48.304600 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:48.327083 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:48.405492 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:48.405497 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:48.415930 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:48.430749 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:48.494919 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:48.494933 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:48.498543 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:48.507103 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:48.576037 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:48.578887 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:48.596406 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:48.620738 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:48.707898 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:48.707926 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:48.711780 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:48.740509 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:48.792544 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:48.794991 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:48.790067 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:48.811043 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:48.862429 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:48.866396 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:48.871430 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:48.882051 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:48.964782 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:48.966307 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:48.970219 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:48.966179 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:49.062819 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:49.060471 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:49.080837 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:49.090893 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:49.143910 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:49.144166 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:49.145859 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:49.153399 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:49.220355 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:49.224639 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:49.230380 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:49.234394 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:49.281416 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:49.283978 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:49.281616 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:49.281158 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:49.330207 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:49.330010 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:49.337595 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:49.337630 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:49.384075 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:49.384809 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:49.384340 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:49.404674 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:49.510068 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:49.510364 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:49.530431 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:49.560476 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0310 20:17:49.605000 281473253340032 failure_handler_test.py:195] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0310 20:17:49.606453 281473253340032 failure_handler_test.py:195] epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I0310 20:17:49.606454 281473253340032 failure_handler_test.py:195] epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0310 20:17:49.608344 281473253340032 failure_handler_test.py:195] epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:49.619457 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:49.627124 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:49.629290 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:49.638491 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:49.730955 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:49.735108 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:49.744472 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:49.800660 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:49.859945 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:49.864434 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:49.864586 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:49.856305 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:49.924571 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:49.934248 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:49.925298 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:49.940402 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:49.998656 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:50.005478 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:50.005939 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:50.019651 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:50.074535 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:50.075532 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:50.080801 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:50.076425 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:50.152538 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:50.155802 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:50.159846 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:50.160172 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:50.217323 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:50.218542 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:50.232119 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:50.237265 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:50.292407 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:50.292926 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:50.305524 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:50.316018 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:50.395726 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:50.408514 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:50.416144 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:50.422234 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:50.472810 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:50.472937 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:50.484985 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:50.491692 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:50.560292 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:50.562040 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:50.562282 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:50.564005 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:50.616699 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:50.617477 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:50.615726 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:50.615087 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:50.670600 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:50.671193 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:50.674190 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:50.690765 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:50.755207 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:50.745564 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:50.761076 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:50.777637 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-3]: I0310 20:17:50.829323 281473253340032 failure_handler_test.py:195] epoch 3 finished [worker-2]: I0310 20:17:50.829555 281473253340032 failure_handler_test.py:195] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-1]: I0310 20:17:50.835457 281473253340032 failure_handler_test.py:195] epoch 3 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:50.838509 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:50.829326 281473253340032 failure_handler_test.py:195] epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:50.838031 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:50.854776 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:50.860708 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:50.930511 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:50.933999 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:50.933474 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:50.934216 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:51.019878 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:51.041007 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:51.057045 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:51.077749 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:51.156342 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:51.168119 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:51.174917 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:51.170562 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:51.229275 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:51.229297 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:51.245356 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:51.254386 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:51.337504 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:51.337523 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:51.351866 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:51.332036 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:51.441357 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:51.424405 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:51.501551 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:51.581458 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:51.697199 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:51.702967 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:51.710615 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:51.700856 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:51.808510 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:51.846257 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:51.832966 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:51.849450 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:52.060310 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:52.063514 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:52.064503 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:52.059897 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:52.128190 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:52.129436 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:52.133113 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:52.152078 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:52.209144 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:52.209499 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:52.208983 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:52.230869 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:52.282023 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:52.284286 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:52.284644 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:52.284637 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:52.370310 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:52.378663 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:52.379898 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:52.376901 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:52.476966 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:52.493069 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:52.495373 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:52.524516 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0310 20:17:52.579885 281473253340032 failure_handler_test.py:195] epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-0]: I0310 20:17:52.580273 281473253340032 failure_handler_test.py:195] epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I0310 20:17:52.581713 281473253340032 failure_handler_test.py:195] epoch 4 finished [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0310 20:17:52.582103 281473253340032 failure_handler_test.py:195] epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:52.590541 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:52.592021 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:52.592546 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:52.594835 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:52.709809 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:52.720697 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:52.726294 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:52.731914 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:52.816830 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:52.826315 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:52.846629 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:52.866274 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:52.947142 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:52.955331 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:52.953090 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:52.972584 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:53.046325 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:53.046360 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:53.051895 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:53.058392 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:53.113619 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:53.115534 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:53.115541 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:53.127424 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:53.196622 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:53.206888 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:53.208355 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:53.218114 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:53.320714 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:53.341624 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:53.371946 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:53.360650 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:53.457408 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:53.461419 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:53.481300 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:53.491327 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:53.570513 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:53.585270 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:53.575583 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:53.599409 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:53.727458 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:53.735174 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:53.747412 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:53.745454 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:53.860488 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:53.866902 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:53.883417 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:53.903247 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:54.019195 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:54.024138 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:54.041964 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:54.051792 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:54.138581 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:54.138919 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:54.152638 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:54.171633 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:54.237375 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:54.237404 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:54.243578 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:54.251944 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 5 finished [worker-1]: INFO:tensorflow:epoch 5 finished [worker-0]: I0310 20:17:54.325174 281473253340032 failure_handler_test.py:195] epoch 5 finished [worker-2]: INFO:tensorflow:epoch 5 finished [worker-2]: I0310 20:17:54.325440 281473253340032 failure_handler_test.py:195] epoch 5 finished [worker-1]: I0310 20:17:54.325363 281473253340032 failure_handler_test.py:195] epoch 5 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:54.335875 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 5 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:54.336710 281473253340032 failure_handler_test.py:195] epoch 5 finished [worker-2]: I0310 20:17:54.337294 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:54.336096 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:54.387398 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:54.470062 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:54.473910 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:54.492004 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:54.504275 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:54.626167 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:54.637034 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:54.642186 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:54.655983 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:54.711456 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:54.713559 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:54.714474 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:54.727566 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:54.809746 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:54.820090 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:54.842439 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:54.877454 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:54.936085 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:54.938544 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:54.938328 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:54.962232 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:55.019958 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:55.022887 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:55.031175 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:55.040830 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:55.162838 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:55.169911 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:55.187742 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:55.165235 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:55.313387 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:55.320853 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:55.332109 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:55.330859 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:55.474899 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:55.478504 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:55.506936 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:55.530729 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:55.632367 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:55.630531 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:55.652148 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:55.671888 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:55.749408 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:55.752892 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:55.754503 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:55.791884 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:55.876129 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:55.876989 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:55.891968 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:55.901791 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:55.983100 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:55.993466 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:55.981988 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:56.002159 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:56.078330 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:56.078113 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:56.078990 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:56.078398 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 6 finished [worker-0]: INFO:tensorflow:epoch 6 finished [worker-3]: I0310 20:17:56.143267 281473253340032 failure_handler_test.py:195] epoch 6 finished [worker-2]: INFO:tensorflow:epoch 6 finished [worker-0]: I0310 20:17:56.143615 281473253340032 failure_handler_test.py:195] epoch 6 finished [worker-1]: INFO:tensorflow:epoch 6 finished [worker-1]: I0310 20:17:56.145768 281473253340032 failure_handler_test.py:195] epoch 6 finished [worker-2]: I0310 20:17:56.145154 281473253340032 failure_handler_test.py:195] epoch 6 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:56.156896 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:56.170500 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:56.171907 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:56.171877 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:56.255566 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:56.259799 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:56.265661 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:56.277448 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:56.362438 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:56.357703 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:56.360719 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:56.360720 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:56.439644 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:56.440999 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:56.439298 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:56.441025 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:56.512514 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:56.512660 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:56.513060 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:56.529207 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:56.585268 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:56.585462 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:56.598068 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:56.607549 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:56.664111 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:56.669337 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:56.667806 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:56.701728 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:56.756638 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:56.756678 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:56.781261 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:56.771954 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:56.868403 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:56.874965 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:56.890504 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:56.932666 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:57.060799 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:57.060674 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:57.070448 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:57.059418 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:57.120659 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:57.121633 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:57.125428 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:57.128207 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:57.195221 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:57.221018 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:57.224617 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:57.236709 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:57.348425 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:57.356591 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:57.349193 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:57.360398 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:57.410544 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:57.430809 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:57.430511 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:57.452422 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:17:57.545607 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:17:57.548445 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:17:57.553529 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:17:57.546295 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 7 finished [worker-0]: I0310 20:17:57.622727 281473253340032 failure_handler_test.py:195] epoch 7 finished [worker-1]: INFO:tensorflow:epoch 7 finished [worker-1]: I0310 20:17:57.623429 281473253340032 failure_handler_test.py:195] epoch 7 finished [worker-0]: INFO:tensorflow:Training finished. [worker-1]: INFO:tensorflow:Training finished. [worker-0]: I0310 20:17:57.624516 281473253340032 failure_handler_test.py:245] Training finished. [worker-1]: I0310 20:17:57.624536 281473253340032 failure_handler_test.py:245] Training finished. [worker-3]: INFO:tensorflow:epoch 7 finished [worker-3]: I0310 20:17:57.626548 281473253340032 failure_handler_test.py:195] epoch 7 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0310 20:17:57.627540 281473253340032 failure_handler_test.py:245] Training finished. [worker-2]: INFO:tensorflow:epoch 7 finished [worker-2]: I0310 20:17:57.632679 281473253340032 failure_handler_test.py:195] epoch 7 finished [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I0310 20:17:57.634106 281473253340032 failure_handler_test.py:245] Training finished. I0310 20:17:58.538911 281473343124352 multi_process_runner.py:646] worker-0 exit code: 0 I0310 20:17:58.539188 281473343124352 multi_process_runner.py:646] worker-1 exit code: 0 I0310 20:17:58.539322 281473343124352 multi_process_runner.py:646] worker-2 exit code: 0 I0310 20:17:58.539443 281473343124352 multi_process_runner.py:646] worker-3 exit code: 0 I0310 20:17:58.541322 281473343124352 multi_process_runner.py:662] Joining log reading threads. I0310 20:17:58.541550 281473343124352 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker): 17.44s I0310 20:17:58.809298 281473343124352 test_util.py:2467] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker): 17.44s [ OK ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 21627 I0310 20:17:58.811135 281473343124352 test_util.py:3797] Using local port 21627 INFO:tensorflow:Using local port 21664 I0310 20:17:58.811632 281473343124352 test_util.py:3797] Using local port 21664 INFO:tensorflow:Using local port 20755 I0310 20:17:58.812030 281473343124352 test_util.py:3797] Using local port 20755 INFO:tensorflow:Using local port 15397 I0310 20:17:58.812416 281473343124352 test_util.py:3797] Using local port 15397 INFO:tensorflow:Cluster starting. I0310 20:17:58.984331 281473343124352 failure_handler_test.py:291] Cluster starting. [worker-0]: I0310 20:17:59.207450 281473253340032 multi_process_runner.py:840] Subprocess with PID 1078313 (worker, 0) is now being started. [worker-0]: I0310 20:17:59.207884 281473253340032 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:21627", "localhost:21664", "localhost:20755", "localhost:15397"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0310 20:17:59.255373 281473253340032 multi_process_runner.py:840] Subprocess with PID 1078383 (worker, 1) is now being started. [worker-2]: I0310 20:17:59.255524 281473253340032 multi_process_runner.py:840] Subprocess with PID 1078390 (worker, 2) is now being started. [worker-2]: I0310 20:17:59.255890 281473253340032 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:21627", "localhost:21664", "localhost:20755", "localhost:15397"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-1]: I0310 20:17:59.255801 281473253340032 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:21627", "localhost:21664", "localhost:20755", "localhost:15397"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-3]: I0310 20:17:59.261040 281473253340032 multi_process_runner.py:840] Subprocess with PID 1078399 (worker, 3) is now being started. [worker-3]: I0310 20:17:59.261472 281473253340032 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:21627", "localhost:21664", "localhost:20755", "localhost:15397"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: 2023-03-10 20:17:59.288737: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:20755 [worker-3]: 2023-03-10 20:17:59.294512: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:15397 [worker-1]: 2023-03-10 20:17:59.312789: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:21664 [worker-0]: 2023-03-10 20:17:59.480506: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:21627 [worker-0]: 2023-03-10 20:17:59.500934: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 3474942304908191290 [worker-0]: 2023-03-10 20:17:59.501705: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 3499067569365090603 [worker-1]: 2023-03-10 20:17:59.505637: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-2]: 2023-03-10 20:17:59.505978: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: 2023-03-10 20:17:59.514597: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 11111577418493316265 [worker-0]: 2023-03-10 20:17:59.514804: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: 2023-03-10 20:18:00.315235: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 1640121155731826410 [worker-3]: 2023-03-10 20:18:00.315465: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0310 20:18:00.318079 281473253340032 collective_all_reduce_strategy.py:532] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0310 20:18:00.319624 281473253340032 collective_all_reduce_strategy.py:532] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0310 20:18:00.318298 281473253340032 collective_all_reduce_strategy.py:532] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0310 20:18:00.322890 281473253340032 collective_all_reduce_strategy.py:532] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Check health not enabled. [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0310 20:18:00.377873 281473253340032 collective_all_reduce_strategy.py:575] Check health not enabled. [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0310 20:18:00.388365 281473253340032 collective_all_reduce_strategy.py:575] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21627', 'localhost:21664', 'localhost:20755', 'localhost:15397']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0310 20:18:00.388657 281473253340032 collective_all_reduce_strategy.py:577] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21627', 'localhost:21664', 'localhost:20755', 'localhost:15397']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0310 20:18:00.372963 281473253340032 collective_all_reduce_strategy.py:575] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21627', 'localhost:21664', 'localhost:20755', 'localhost:15397']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Check health not enabled. [worker-0]: I0310 20:18:00.373503 281473253340032 collective_all_reduce_strategy.py:577] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21627', 'localhost:21664', 'localhost:20755', 'localhost:15397']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0310 20:18:00.423493 281473253340032 collective_all_reduce_strategy.py:575] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21627', 'localhost:21664', 'localhost:20755', 'localhost:15397']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0310 20:18:00.423943 281473253340032 collective_all_reduce_strategy.py:577] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21627', 'localhost:21664', 'localhost:20755', 'localhost:15397']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21627', 'localhost:21664', 'localhost:20755', 'localhost:15397']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0310 20:18:00.378249 281473253340032 collective_all_reduce_strategy.py:577] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:21627', 'localhost:21664', 'localhost:20755', 'localhost:15397']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0310 20:18:00.511637 281473253340032 failure_handling.py:633] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I0310 20:18:00.512477 281473253340032 failure_handling.py:673] Start watcher for local signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0310 20:18:00.515987 281473253340032 failure_handling.py:633] Start watcher for peer's signal. [worker-1]: I0310 20:18:00.511924 281473253340032 failure_handling.py:633] Start watcher for peer's signal. [worker-3]: I0310 20:18:00.512689 281473253340032 failure_handling.py:537] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0310 20:18:00.512970 281473253340032 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0310 20:18:00.513129 281473253340032 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0310 20:18:00.530020 281473253340032 failure_handling.py:673] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0310 20:18:00.530305 281473253340032 failure_handling.py:537] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0310 20:18:00.530599 281473253340032 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0310 20:18:00.530760 281473253340032 failure_handler_test.py:197] Start training at 0 [worker-0]: I0310 20:18:00.506278 281473253340032 failure_handling.py:633] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I0310 20:18:00.510828 281473253340032 failure_handling.py:673] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0310 20:18:00.511220 281473253340032 failure_handling.py:537] PreemptionCheckpointHandler initialized or restored. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I0310 20:18:00.566292 281473253340032 failure_handling.py:673] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0310 20:18:00.566717 281473253340032 failure_handling.py:537] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0310 20:18:00.567056 281473253340032 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0310 20:18:00.567222 281473253340032 failure_handler_test.py:197] Start training at 0 [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0310 20:18:00.511538 281473253340032 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0310 20:18:00.511706 281473253340032 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:00.730081 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:00.731044 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:00.736485 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:00.743566 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending sigterm [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I0310 20:18:03.109219 281473343124352 failure_handler_test.py:296] sending sigterm INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker): 13.81s [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I0310 20:18:12.624700 281473343124352 test_util.py:2467] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker): 13.81s [worker-2]: I0310 20:18:00.844441 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:00.845927 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:00.845121 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:00.898057 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:00.896631 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [ FAILED ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 ====================================================================== ERROR: test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker (__main__.PreemptionCheckpointTest) PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker(api_wrapping_train=True, input_arg='manager', strategy_option='MWMS_multi_worker') ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 298, in test_preemption_checkpointing os.kill(mpr.get_process_id('worker', killed_worker), signal.SIGTERM) ProcessLookupError: [Errno 3] No such process ---------------------------------------------------------------------- [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:00.948450 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 Ran 4 tests in 41.303s [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:00.999334 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff95632af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0310 20:18:01.041356 281473253340032 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff95632af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: I0310 20:18:00.948291 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:00.845532 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 FAILED (errors=1) [worker-1]: I0310 20:18:01.051636 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff956410d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: I0310 20:18:00.999241 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:00.896487 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: W0310 20:18:01.092810 281473253340032 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff956410d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: I0310 20:18:00.946749 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff95631af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0310 20:18:01.041073 281473253340032 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff95631af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:01.205507 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:00.998175 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff95632af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: I0310 20:18:01.052251 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:01.277793 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: W0310 20:18:01.041236 281473253340032 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff95632af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff956400d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.050590 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:01.323576 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: W0310 20:18:01.092603 281473253340032 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff956400d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff956420d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: W0310 20:18:01.092695 281473253340032 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff956420d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: I0310 20:18:01.369168 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:01.230245 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.204212 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:01.276891 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.276887 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:01.323484 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.322846 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.368878 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.415751 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:00.897380 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.459733 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:01.369132 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:00.948614 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:00.999521 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff95632af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:01.414659 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: W0310 20:18:01.041445 281473253340032 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff95632af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: I0310 20:18:01.504401 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:01.414311 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:01.459848 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.051789 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:01.504423 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff956410d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.549327 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: W0310 20:18:01.092902 281473253340032 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff956410d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: I0310 20:18:01.549465 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:01.459680 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:01.593661 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.593583 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:01.504023 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.631466 281473253340032 failure_handler_test.py:195] epoch 0 finished [worker-1]: I0310 20:18:01.631583 281473253340032 failure_handler_test.py:195] epoch 0 finished [worker-3]: I0310 20:18:01.549200 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:01.639874 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.639556 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:01.593473 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:01.684329 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 0 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:01.728557 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:01.775032 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.684241 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:01.631334 281473253340032 failure_handler_test.py:195] epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:01.639394 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:01.684071 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.228672 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:01.820284 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.728582 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:01.728364 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.277383 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:01.865651 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.775017 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.322757 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:01.911391 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.820107 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:01.774825 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.368794 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:01.819961 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.865838 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.414423 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.459553 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.911242 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:01.865373 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.504367 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:01.911203 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:01.956735 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.549396 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:01.956535 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.001471 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.593563 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.046547 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.631583 281473253340032 failure_handler_test.py:195] epoch 0 finished [worker-3]: I0310 20:18:02.001729 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.091179 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.639525 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.136198 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.046358 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.091060 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.181119 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.684235 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.225827 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.728568 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.774983 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.269855 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.820062 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.307662 281473253340032 failure_handler_test.py:195] epoch 1 finished [worker-2]: I0310 20:18:01.865800 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.317044 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.911221 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.362935 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:01.956628 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.407385 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.452383 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.001493 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.135792 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.497118 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.046481 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.180943 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:01.956715 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.091159 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.001760 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.542011 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.136104 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.046371 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.181163 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.091299 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.586899 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.136190 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.225363 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.631248 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.225059 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.269826 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.181118 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.676129 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.225276 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.269662 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.307830 281473253340032 failure_handler_test.py:195] epoch 1 finished [worker-0]: I0310 20:18:02.720858 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.307511 281473253340032 failure_handler_test.py:195] epoch 1 finished [worker-2]: I0310 20:18:02.317370 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.765400 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.269920 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 1 finished [worker-3]: I0310 20:18:02.317512 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.307789 281473253340032 failure_handler_test.py:195] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.362859 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.317480 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.363084 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.407705 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.407922 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.362913 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.452337 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.810924 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.496925 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.857771 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.452367 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.541854 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.903247 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.407385 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.497138 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.947160 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.542082 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.586998 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.586772 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.452291 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.631344 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.984516 281473253340032 failure_handler_test.py:195] epoch 2 finished [worker-3]: I0310 20:18:02.631154 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.497114 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.676185 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.541952 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.720885 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.586900 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.765599 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.631146 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.811033 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.675987 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:02.992144 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.857945 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.676066 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.903033 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.036307 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.720608 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.720839 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.947295 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.080347 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.765300 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.765330 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.124201 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.984598 281473253340032 failure_handler_test.py:195] epoch 2 finished [worker-3]: I0310 20:18:02.810815 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.810904 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:02.992234 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.168823 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.857789 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.857889 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.036426 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.903197 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.212859 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.902777 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.080449 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.947122 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.947078 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 2 finished [worker-0]: I0310 20:18:03.257234 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-1]: I0310 20:18:03.124366 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.984343 281473253340032 failure_handler_test.py:195] epoch 2 finished [worker-2]: I0310 20:18:02.984625 281473253340032 failure_handler_test.py:195] epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.301800 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.168915 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:02.992111 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:02.991927 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.036261 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.212841 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.036142 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.346655 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.257826 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.080351 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.393023 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.124166 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.438338 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.168778 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.482951 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.212866 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.527332 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.257162 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.301826 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.572860 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.301904 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.346422 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.617891 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.346221 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.393216 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 3 finished [worker-2]: I0310 20:18:03.393079 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.655232 281473253340032 failure_handler_test.py:195] epoch 3 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.438679 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.438508 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.663035 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.483023 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.482928 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.707360 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.527396 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.527316 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.752010 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.572794 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.572822 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.797291 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.617933 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.617862 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.842675 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 3 finished [worker-1]: I0310 20:18:03.655307 281473253340032 failure_handler_test.py:195] epoch 3 finished [worker-0]: I0310 20:18:03.887990 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.655383 281473253340032 failure_handler_test.py:195] epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.663068 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.933841 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.663162 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.707505 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:03.978725 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.707436 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.752068 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.023499 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.751908 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.797308 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.069361 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.797294 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.842640 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.115200 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.843880 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.888101 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.160263 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.888193 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.934029 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.080176 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.204888 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.933833 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.249300 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:03.978763 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:03.978679 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.124154 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.168629 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.294655 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.023430 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.023576 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 4 finished [worker-2]: I0310 20:18:04.069340 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.212512 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.332815 281473253340032 failure_handler_test.py:195] epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.256928 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.115190 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.069400 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.340584 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.160118 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.301744 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.115304 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.385755 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.204886 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.346223 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.160234 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.430311 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.249365 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.393115 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.204944 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.477370 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.294599 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.438273 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.249438 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.523673 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.332998 281473253340032 failure_handler_test.py:195] epoch 4 finished [worker-0]: I0310 20:18:04.569847 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.482933 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.294260 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.527162 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.340510 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.616083 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.332951 281473253340032 failure_handler_test.py:195] epoch 4 finished [worker-2]: I0310 20:18:04.385767 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.659926 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.572993 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.430367 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.617691 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.340549 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.705256 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.477463 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.655020 281473253340032 failure_handler_test.py:195] epoch 3 finished [worker-1]: I0310 20:18:04.385473 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.750266 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.523616 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.430332 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.662782 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.794509 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.569549 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.476924 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.707342 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.838741 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.522843 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.568952 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.883124 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.751895 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.615769 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.797118 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.660676 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.928335 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.705585 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:04.972768 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.842254 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 5 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.750338 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.011163 281473253340032 failure_handler_test.py:195] epoch 5 finished [worker-3]: I0310 20:18:03.887874 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.794745 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.020531 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.933737 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.838963 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.065296 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:03.979079 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.883226 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.109934 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.023355 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.928495 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.154140 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.069040 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:04.972797 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.199528 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.616061 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.660005 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.705287 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.115047 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.750211 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.794517 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.160038 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.838831 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.204771 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 5 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.011260 281473253340032 failure_handler_test.py:195] epoch 5 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.883143 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.244722 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.249021 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.294147 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:04.928360 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.332665 281473253340032 failure_handler_test.py:195] epoch 4 finished [worker-2]: I0310 20:18:04.972762 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 5 finished [worker-3]: I0310 20:18:04.340240 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.011335 281473253340032 failure_handler_test.py:195] epoch 5 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.385332 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.020389 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.020730 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.289170 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.065281 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.334189 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.109953 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.379596 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.154304 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.424494 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.199491 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.469674 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.244712 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.513935 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.289239 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.560232 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.334190 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.605951 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.379564 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.650776 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.424468 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 6 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.688977 281473253340032 failure_handler_test.py:195] epoch 6 finished [worker-2]: I0310 20:18:05.469647 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.696525 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.513982 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.741493 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.560251 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.786136 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.605978 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.830291 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.650804 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.875193 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 6 finished [worker-0]: I0310 20:18:05.921058 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.689158 281473253340032 failure_handler_test.py:195] epoch 6 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.066082 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:05.966417 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.696818 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.109962 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.154248 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:06.012388 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.741465 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.199795 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.785995 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.244822 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.289132 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.334452 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.379972 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.424455 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.469670 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.514182 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.560144 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.606241 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.650971 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 6 finished [worker-1]: I0310 20:18:05.689113 281473253340032 failure_handler_test.py:195] epoch 6 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:06.057363 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.830270 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.875291 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:06.104070 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.920995 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.696543 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:06.149463 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:05.966435 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.741345 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:06.194478 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:06.012420 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.786321 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:06.238490 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:06.057346 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.830334 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:06.283109 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:06.104422 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:05.875855 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:06.328006 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:06.149461 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 7 finished [worker-1]: I0310 20:18:05.921510 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:06.365407 281473253340032 failure_handler_test.py:195] epoch 7 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:06.194258 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Training finished. [worker-1]: I0310 20:18:05.966984 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0310 20:18:06.366500 281473253340032 failure_handler_test.py:245] Training finished. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:06.238516 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:06.012774 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:06.057540 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:06.283102 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:06.104611 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:06.327955 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 7 finished [worker-1]: I0310 20:18:06.149684 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:06.365601 281473253340032 failure_handler_test.py:195] epoch 7 finished [worker-1]: I0310 20:18:06.194331 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Training finished. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0310 20:18:06.366604 281473253340032 failure_handler_test.py:245] Training finished. [worker-1]: I0310 20:18:06.238955 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:06.283314 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0310 20:18:06.327972 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 7 finished [worker-1]: I0310 20:18:06.365545 281473253340032 failure_handler_test.py:195] epoch 7 finished [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I0310 20:18:06.366804 281473253340032 failure_handler_test.py:245] Training finished. [worker-3]: I0310 20:18:04.430179 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.476615 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.522693 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.568648 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.615553 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.660052 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.705399 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.750167 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.794594 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.838731 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.882973 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.928199 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:04.972483 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 5 finished [worker-3]: I0310 20:18:05.010997 281473253340032 failure_handler_test.py:195] epoch 5 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.020114 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.065505 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.109788 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.154107 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.199824 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.244666 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.288953 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.334247 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.379772 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.424235 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.469236 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.513904 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.559702 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.606130 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.650820 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 6 finished [worker-3]: I0310 20:18:05.688718 281473253340032 failure_handler_test.py:195] epoch 6 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.696186 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.741199 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.785877 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.830058 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.875601 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.921442 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:05.966746 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:06.013108 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:06.057206 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:06.104525 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:06.149426 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:06.193925 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:06.238757 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:06.282932 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0310 20:18:06.327743 281473253340032 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 7 finished [worker-3]: I0310 20:18:06.365188 281473253340032 failure_handler_test.py:195] epoch 7 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0310 20:18:06.366402 281473253340032 failure_handler_test.py:245] Training finished. ================================================================================ ==================== Test output for //tensorflow/dtensor/python/tests:input_util_test (shard 4 of 8): Running tests under Python 3.9.16: /usr/local/bin/python3 [ RUN ] DTensorDatasetTest.testIterOnBatchedDatasetEager I0310 20:18:33.399154 281473157723008 mesh_util.py:35] This is client 0 of 1 clients I0310 20:18:33.399442 281473157723008 mesh_util.py:36] Number of global CPU devices: 16 I0310 20:18:33.399610 281473157723008 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0310 20:18:33.400213 281473157723008 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0310 20:18:33.400406 281473157723008 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-03-10 20:18:33.774609: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:33.849605: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:33.918405: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:33.987343: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:34.086359: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:34.087284: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:34.171365: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:34.172447: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:34.286036: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:34.287055: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:34.359609: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:34.360665: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:34.488628: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:34.489499: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:34.580059: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:34.580967: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:34.662392: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:34.663369: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:34.725467: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:34.726418: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:34.847275: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:34.848178: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:34.908084: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:34.908992: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:34.975161: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:34.976090: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:35.037897: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:35.038880: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:35.110468: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:35.111414: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:35.180525: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:35.181523: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:35.248679: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:35.249679: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:35.328061: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:35.328997: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:35.609642: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [1] [[{{node Placeholder/_1}}]] INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterOnBatchedDatasetEager): 2.79s I0310 20:18:36.183363 281473157723008 test_util.py:2467] time(__main__.DTensorDatasetTest.testIterOnBatchedDatasetEager): 2.79s [ OK ] DTensorDatasetTest.testIterOnBatchedDatasetEager [ RUN ] DTensorDatasetTest.testIterTupleInputsEager I0310 20:18:36.185537 281473157723008 mesh_util.py:35] This is client 0 of 1 clients I0310 20:18:36.185711 281473157723008 mesh_util.py:36] Number of global CPU devices: 16 I0310 20:18:36.185870 281473157723008 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0310 20:18:36.186362 281473157723008 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0310 20:18:36.186509 281473157723008 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-03-10 20:18:36.457473: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:36.527837: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:36.595593: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:36.663429: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:36.718708: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:36.719615: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:36.785592: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:36.786536: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:36.855929: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:36.856942: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:36.934117: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:36.935081: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:37.004956: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:37.005847: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:37.071433: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:37.072343: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:37.139609: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:37.140567: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:37.215042: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:37.215988: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:37.586707: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:37.587618: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:38.042760: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:38.043682: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:38.123680: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:38.124608: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:38.195841: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:38.196835: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:38.313691: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:38.314610: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:38.373303: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:38.374199: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:38.509005: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:38.509997: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:38.575446: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:38.576452: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:38.844864: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [1] [[{{node Placeholder/_1}}]] INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterTupleInputsEager): 3.52s I0310 20:18:39.707153 281473157723008 test_util.py:2467] time(__main__.DTensorDatasetTest.testIterTupleInputsEager): 3.52s [ OK ] DTensorDatasetTest.testIterTupleInputsEager [ RUN ] DTensorDatasetTest.testIterWithLayouts4 (images_sharding=['unsharded', 'height', 'width', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=False) I0310 20:18:39.708902 281473157723008 mesh_util.py:35] This is client 0 of 1 clients I0310 20:18:39.709112 281473157723008 mesh_util.py:36] Number of global CPU devices: 16 I0310 20:18:39.709274 281473157723008 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0310 20:18:39.709777 281473157723008 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0310 20:18:39.709928 281473157723008 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-03-10 20:18:40.098844: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:40.201864: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:40.202800: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:40.432399: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:40.433388: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:40.555174: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:40.556235: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:40.655145: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:40.656138: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:40.737653: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:40.738638: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:40.807975: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:40.808932: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:40.897622: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:40.898615: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:40.966703: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:40.967694: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:41.100360: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:41.101359: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:41.171151: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:41.172121: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:41.276904: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:41.277827: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:41.351918: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:41.352847: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:41.481128: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:41.482103: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:41.557244: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:41.558165: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:41.620433: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:41.621367: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:41.725307: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:41.726332: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:41.951529: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [1] [[{{node Placeholder/_1}}]] INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterWithLayouts4 (images_sharding=['unsharded', 'height', 'width', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=False)): 2.94s I0310 20:18:42.650971 281473157723008 test_util.py:2467] time(__main__.DTensorDatasetTest.testIterWithLayouts4 (images_sharding=['unsharded', 'height', 'width', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=False)): 2.94s [ OK ] DTensorDatasetTest.testIterWithLayouts4 (images_sharding=['unsharded', 'height', 'width', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=False) [ RUN ] DTensorDatasetTest.testRangeIterationEager I0310 20:18:42.652782 281473157723008 mesh_util.py:35] This is client 0 of 1 clients I0310 20:18:42.652968 281473157723008 mesh_util.py:36] Number of global CPU devices: 16 I0310 20:18:42.653132 281473157723008 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0310 20:18:42.653602 281473157723008 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0310 20:18:42.653749 281473157723008 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-03-10 20:18:42.896465: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_0}}]] 2023-03-10 20:18:43.000657: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:43.057712: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:43.111162: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:43.164393: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:43.208551: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:43.209358: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:43.273302: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:43.274100: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:43.338092: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:43.338870: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:43.405111: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:43.405956: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:43.467669: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:43.468441: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:43.579891: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:43.580652: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:43.643769: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:43.644557: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:43.715663: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:43.716518: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:43.773427: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:43.774202: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:43.829414: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:43.830149: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:43.884441: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:43.885195: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:43.939745: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:43.940533: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:44.014497: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:44.015296: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:44.073726: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:44.074552: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:44.150058: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:44.151107: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:44.260612: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:44.261450: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:44.612931: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:11] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled [[{{node tf.StatefulPartitionedCall/eager_operation}}]] 2023-03-10 20:18:44.615324: E tensorflow/dtensor/cc/dtensor_device.cc:2032] Encountered error while executing function: CopyToMesh__func_5950845106898307211_6858324592425701659_10 for mesh : |batch=4,height=2,width=2|0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15|0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15|/job:localhost/replica:0/task:0/device:CPU:0,/job:localhost/replica:0/task:0/device:CPU:1,/job:localhost/replica:0/task:0/device:CPU:2,/job:localhost/replica:0/task:0/device:CPU:3,/job:localhost/replica:0/task:0/device:CPU:4,/job:localhost/replica:0/task:0/device:CPU:5,/job:localhost/replica:0/task:0/device:CPU:6,/job:localhost/replica:0/task:0/device:CPU:7,/job:localhost/replica:0/task:0/device:CPU:8,/job:localhost/replica:0/task:0/device:CPU:9,/job:localhost/replica:0/task:0/device:CPU:10,/job:localhost/replica:0/task:0/device:CPU:11,/job:localhost/replica:0/task:0/device:CPU:12,/job:localhost/replica:0/task:0/device:CPU:13,/job:localhost/replica:0/task:0/device:CPU:14,/job:localhost/replica:0/task:0/device:CPU:15 / error : {{function_node IteratorGetNext__func_12989052755378955192_1628789561242421788_9}} End of sequence [[{{node tf.StatefulPartitionedCall/eager_operation}}]] Encountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors. 2023-03-10 20:18:44.616811: E tensorflow/dtensor/cc/dtensor_device.cc:2078] Error executing CopyToMesh {{function_node IteratorGetNext__func_12989052755378955192_1628789561242421788_9}} End of sequence [[{{node tf.StatefulPartitionedCall/eager_operation}}]] Encountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors. INFO:tensorflow:time(__main__.DTensorDatasetTest.testRangeIterationEager): 2.1s I0310 20:18:44.751537 281473157723008 test_util.py:2467] time(__main__.DTensorDatasetTest.testRangeIterationEager): 2.1s [ FAILED ] DTensorDatasetTest.testRangeIterationEager [ RUN ] InputUtilHelpersTest.testShardCounts2 (mesh_dims=[('batch', 8)], layout_specs=['batch'], batch_dim='batch', counts=[1]) I0310 20:18:44.755142 281473157723008 mesh_util.py:35] This is client 0 of 1 clients I0310 20:18:44.755386 281473157723008 mesh_util.py:36] Number of global CPU devices: 8 I0310 20:18:44.755545 281473157723008 mesh_util.py:39] Global device IDs: [0 1 2 3 4 5 6 7] I0310 20:18:44.755933 281473157723008 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7] I0310 20:18:44.756073 281473157723008 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7'] INFO:tensorflow:time(__main__.InputUtilHelpersTest.testShardCounts2 (mesh_dims=[('batch', 8)], layout_specs=['batch'], batch_dim='batch', counts=[1])): 0.0s I0310 20:18:44.756430 281473157723008 test_util.py:2467] time(__main__.InputUtilHelpersTest.testShardCounts2 (mesh_dims=[('batch', 8)], layout_specs=['batch'], batch_dim='batch', counts=[1])): 0.0s [ OK ] InputUtilHelpersTest.testShardCounts2 (mesh_dims=[('batch', 8)], layout_specs=['batch'], batch_dim='batch', counts=[1]) ====================================================================== ERROR: testRangeIterationEager (__main__.DTensorDatasetTest) DTensorDatasetTest.testRangeIterationEager testRangeIterationEager(False) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/tests/input_util_test.py", line 154, in testRangeIteration self.assertDTensorEqual(output, images_layout, d_output) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/tests/test_util.py", line 299, in assertDTensorEqual unpacked = [t.numpy() for t in api.unpack(result_dtensor)] File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/tests/test_util.py", line 299, in unpacked = [t.numpy() for t in api.unpack(result_dtensor)] File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 1211, in numpy maybe_arr = self._numpy() # pylint: disable=protected-access File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 1179, in _numpy raise core._status_to_exception(e) from None # pylint: disable=protected-access tensorflow.python.framework.errors_impl.CancelledError: {{function_node AddV2__func_13230101824050524202_2386512056599665587_11}} Function was cancelled before it was started ---------------------------------------------------------------------- Ran 5 tests in 11.359s FAILED (errors=1) ================================================================================ ==================== Test output for //tensorflow/dtensor/python/tests:input_util_test (shard 4 of 8): Running tests under Python 3.9.16: /usr/local/bin/python3 [ RUN ] DTensorDatasetTest.testIterOnBatchedDatasetEager I0310 20:18:48.218169 281473101427584 mesh_util.py:35] This is client 0 of 1 clients I0310 20:18:48.218406 281473101427584 mesh_util.py:36] Number of global CPU devices: 16 I0310 20:18:48.218565 281473101427584 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0310 20:18:48.219101 281473101427584 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0310 20:18:48.219287 281473101427584 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-03-10 20:18:48.526083: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:48.636048: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:48.696796: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:48.779911: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:48.836090: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:48.837028: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:48.930782: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:48.931703: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:49.091296: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:49.092230: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:49.223832: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:49.224798: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:49.289822: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:49.290710: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:49.352777: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:49.353681: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:49.453416: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:49.454360: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:49.516807: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:49.517776: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:49.584013: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:49.584974: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:49.650981: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:49.651977: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:49.772853: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:49.773823: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:49.898843: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:49.899851: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:49.964024: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:49.964995: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:50.051367: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:50.052356: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:50.272649: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:50.273637: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:50.347724: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:50.348654: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:50.639900: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [1] [[{{node Placeholder/_1}}]] INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterOnBatchedDatasetEager): 3.02s I0310 20:18:51.237343 281473101427584 test_util.py:2467] time(__main__.DTensorDatasetTest.testIterOnBatchedDatasetEager): 3.02s [ OK ] DTensorDatasetTest.testIterOnBatchedDatasetEager [ RUN ] DTensorDatasetTest.testIterTupleInputsEager I0310 20:18:51.239509 281473101427584 mesh_util.py:35] This is client 0 of 1 clients I0310 20:18:51.239691 281473101427584 mesh_util.py:36] Number of global CPU devices: 16 I0310 20:18:51.239844 281473101427584 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0310 20:18:51.240284 281473101427584 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0310 20:18:51.240423 281473101427584 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-03-10 20:18:51.454249: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:51.515785: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:51.575705: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:51.637452: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:51.689907: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:51.690827: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:51.808898: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:51.809889: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:51.907545: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:51.908509: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:51.978844: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:51.979853: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:52.069551: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:52.070495: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:52.135948: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:52.137045: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:52.210541: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:52.211524: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:52.354732: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:52.355711: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:52.427112: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:52.428097: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:52.497327: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:52.498326: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:52.585999: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:52.587505: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:52.672087: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:52.673071: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:52.739702: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:52.740675: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:52.805245: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:52.806269: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:52.871435: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:52.872438: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:52.938669: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:52.939646: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:53.287615: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [1] [[{{node Placeholder/_1}}]] INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterTupleInputsEager): 2.51s I0310 20:18:53.753554 281473101427584 test_util.py:2467] time(__main__.DTensorDatasetTest.testIterTupleInputsEager): 2.51s [ OK ] DTensorDatasetTest.testIterTupleInputsEager [ RUN ] DTensorDatasetTest.testIterWithLayouts4 (images_sharding=['unsharded', 'height', 'width', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=False) I0310 20:18:53.755147 281473101427584 mesh_util.py:35] This is client 0 of 1 clients I0310 20:18:53.755316 281473101427584 mesh_util.py:36] Number of global CPU devices: 16 I0310 20:18:53.755465 281473101427584 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0310 20:18:53.755902 281473101427584 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0310 20:18:53.756040 281473101427584 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-03-10 20:18:53.977323: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:54.037339: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:54.038319: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:54.143305: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:54.144307: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:54.228674: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:54.229714: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:54.295911: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:54.296908: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:54.366607: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:54.367590: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:54.442784: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:54.443796: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:54.548943: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:54.549909: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:54.611746: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:54.612713: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:54.679897: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:54.680916: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:54.793238: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:54.794255: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:55.010665: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:55.011637: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:55.079088: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:55.080073: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:55.196786: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:55.197789: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-03-10 20:18:55.406036: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:55.407063: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:55.538722: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:55.539724: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:55.618437: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-03-10 20:18:55.619424: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-03-10 20:18:55.976666: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [1] [[{{node Placeholder/_1}}]] INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterWithLayouts4 (images_sharding=['unsharded', 'height', 'width', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=False)): 2.79s I0310 20:18:56.542800 281473101427584 test_util.py:2467] time(__main__.DTensorDatasetTest.testIterWithLayouts4 (images_sharding=['unsharded', 'height', 'width', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=False)): 2.79s [ OK ] DTensorDatasetTest.testIterWithLayouts4 (images_sharding=['unsharded', 'height', 'width', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=False) [ RUN ] DTensorDatasetTest.testRangeIterationEager I0310 20:18:56.544351 281473101427584 mesh_util.py:35] This is client 0 of 1 clients I0310 20:18:56.544497 281473101427584 mesh_util.py:36] Number of global CPU devices: 16 I0310 20:18:56.544651 281473101427584 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0310 20:18:56.545083 281473101427584 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0310 20:18:56.545222 281473101427584 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-03-10 20:18:56.819111: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_0}}]] 2023-03-10 20:18:56.868391: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:56.949191: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:57.028643: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:57.084427: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-03-10 20:18:57.128484: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:57.129264: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:57.189838: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:57.190639: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:57.254458: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:57.255302: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:57.319559: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:57.320421: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:57.385016: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:57.385790: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:57.659271: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:57.660041: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:57.722400: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:57.723227: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:57.794600: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:57.795433: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:57.853376: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:57.854147: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:57.957756: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:57.958520: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:58.016355: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:58.017090: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:58.076102: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:58.076904: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:58.142806: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:58.143620: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:58.207590: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:58.208386: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:58.266883: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:58.267693: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-03-10 20:18:58.353724: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:58.354529: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-03-10 20:18:58.834881: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:11] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled [[{{node tf.StatefulPartitionedCall/eager_operation}}]] 2023-03-10 20:18:58.835498: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:2] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled [[{{node tf.StatefulPartitionedCall/eager_operation}}]] 2023-03-10 20:18:58.837364: E tensorflow/dtensor/cc/dtensor_device.cc:2032] Encountered error while executing function: CopyToMesh__func_5950845106898307211_6858324592425701659_10 for mesh : |batch=4,height=2,width=2|0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15|0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15|/job:localhost/replica:0/task:0/device:CPU:0,/job:localhost/replica:0/task:0/device:CPU:1,/job:localhost/replica:0/task:0/device:CPU:2,/job:localhost/replica:0/task:0/device:CPU:3,/job:localhost/replica:0/task:0/device:CPU:4,/job:localhost/replica:0/task:0/device:CPU:5,/job:localhost/replica:0/task:0/device:CPU:6,/job:localhost/replica:0/task:0/device:CPU:7,/job:localhost/replica:0/task:0/device:CPU:8,/job:localhost/replica:0/task:0/device:CPU:9,/job:localhost/replica:0/task:0/device:CPU:10,/job:localhost/replica:0/task:0/device:CPU:11,/job:localhost/replica:0/task:0/device:CPU:12,/job:localhost/replica:0/task:0/device:CPU:13,/job:localhost/replica:0/task:0/device:CPU:14,/job:localhost/replica:0/task:0/device:CPU:15 / error : {{function_node IteratorGetNext__func_12989052755378955192_1628789561242421788_9}} End of sequence [[{{node tf.StatefulPartitionedCall/eager_operation}}]] Encountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors. 2023-03-10 20:18:58.843011: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:4] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled [[{{node tf.StatefulPartitionedCall/eager_operation}}]] 2023-03-10 20:18:58.846308: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:10] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled [[{{node tf.StatefulPartitionedCall/eager_operation}}]] 2023-03-10 20:18:58.846688: E tensorflow/dtensor/cc/dtensor_device.cc:2078] Error executing CopyToMesh {{function_node IteratorGetNext__func_12989052755378955192_1628789561242421788_9}} End of sequence [[{{node tf.StatefulPartitionedCall/eager_operation}}]] Encountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors. INFO:tensorflow:time(__main__.DTensorDatasetTest.testRangeIterationEager): 2.39s I0310 20:18:58.931040 281473101427584 test_util.py:2467] time(__main__.DTensorDatasetTest.testRangeIterationEager): 2.39s [ FAILED ] DTensorDatasetTest.testRangeIterationEager [ RUN ] InputUtilHelpersTest.testShardCounts2 (mesh_dims=[('batch', 8)], layout_specs=['batch'], batch_dim='batch', counts=[1]) I0310 20:18:58.934089 281473101427584 mesh_util.py:35] This is client 0 of 1 clients I0310 20:18:58.934273 281473101427584 mesh_util.py:36] Number of global CPU devices: 8 I0310 20:18:58.934422 281473101427584 mesh_util.py:39] Global device IDs: [0 1 2 3 4 5 6 7] I0310 20:18:58.934762 281473101427584 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7] I0310 20:18:58.934897 281473101427584 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7'] INFO:tensorflow:time(__main__.InputUtilHelpersTest.testShardCounts2 (mesh_dims=[('batch', 8)], layout_specs=['batch'], batch_dim='batch', counts=[1])): 0.0s I0310 20:18:58.935162 281473101427584 test_util.py:2467] time(__main__.InputUtilHelpersTest.testShardCounts2 (mesh_dims=[('batch', 8)], layout_specs=['batch'], batch_dim='batch', counts=[1])): 0.0s [ OK ] InputUtilHelpersTest.testShardCounts2 (mesh_dims=[('batch', 8)], layout_specs=['batch'], batch_dim='batch', counts=[1]) ====================================================================== ERROR: testRangeIterationEager (__main__.DTensorDatasetTest) DTensorDatasetTest.testRangeIterationEager testRangeIterationEager(False) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/tests/input_util_test.py", line 154, in testRangeIteration self.assertDTensorEqual(output, images_layout, d_output) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/tests/test_util.py", line 299, in assertDTensorEqual unpacked = [t.numpy() for t in api.unpack(result_dtensor)] File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/tests/test_util.py", line 299, in unpacked = [t.numpy() for t in api.unpack(result_dtensor)] File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 1211, in numpy maybe_arr = self._numpy() # pylint: disable=protected-access File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 1179, in _numpy raise core._status_to_exception(e) from None # pylint: disable=protected-access tensorflow.python.framework.errors_impl.CancelledError: {{function_node AddV2__func_13230101824050524202_2386512056599665587_11}} Function was cancelled before it was started ---------------------------------------------------------------------- Ran 5 tests in 10.719s FAILED (errors=1) ================================================================================ ==================== Test output for //tensorflow/compiler/xla/pjrt/distributed:client_server_test: [==========] Running 32 tests from 1 test suite. [----------] Global test environment set-up. [----------] 32 tests from ClientServerTests/ClientServerTest [ RUN ] ClientServerTests/ClientServerTest.ConnectAndShutdownAreBarriers/CoordinationService 2023-03-10 20:20:28.742918: I tensorflow/compiler/xla/pjrt/distributed/service.cc:522] Experimental coordination service is enabled. 2023-03-10 20:20:28.786631: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:0 has connected to coordination service. Incarnation: 6372145913139224795 2023-03-10 20:20:28.796916: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:1 has connected to coordination service. Incarnation: 14763918776491392301 2023-03-10 20:20:28.797046: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:2 has connected to coordination service. Incarnation: 8163593029854639186 2023-03-10 20:20:28.797230: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:28.816147: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:28.816448: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:28.836575: I tensorflow/compiler/xla/pjrt/distributed/client.cc:508] Connected to distributed JAX controller 2023-03-10 20:20:28.847038: I tensorflow/compiler/xla/pjrt/distributed/client.cc:508] Connected to distributed JAX controller 2023-03-10 20:20:28.856167: I tensorflow/compiler/xla/pjrt/distributed/client.cc:508] Connected to distributed JAX controller 2023-03-10 20:20:28.856232: I tensorflow/compiler/xla/pjrt/distributed/client.cc:516] Distributed task shutdown initiated. 2023-03-10 20:20:28.856247: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:464] Coordination agent has initiated Shutdown(). 2023-03-10 20:20:28.856265: I tensorflow/compiler/xla/pjrt/distributed/client.cc:516] Distributed task shutdown initiated. 2023-03-10 20:20:28.856305: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:464] Coordination agent has initiated Shutdown(). 2023-03-10 20:20:28.856405: I tensorflow/compiler/xla/pjrt/distributed/client.cc:516] Distributed task shutdown initiated. 2023-03-10 20:20:28.856436: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:464] Coordination agent has initiated Shutdown(). 2023-03-10 20:20:28.856597: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:1136] Shutdown barrier in coordination service has passed. 2023-03-10 20:20:28.856639: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:630] /job:jax_worker/replica:0/task:2 has disconnected from coordination service. 2023-03-10 20:20:28.856652: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:630] /job:jax_worker/replica:0/task:0 has disconnected from coordination service. 2023-03-10 20:20:28.856664: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:630] /job:jax_worker/replica:0/task:1 has disconnected from coordination service. 2023-03-10 20:20:28.856958: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:483] Coordination agent has successfully shut down. 2023-03-10 20:20:28.856998: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:483] Coordination agent has successfully shut down. 2023-03-10 20:20:28.857188: I tensorflow/compiler/xla/pjrt/distributed/client.cc:518] Distributed task shutdown result: OK 2023-03-10 20:20:28.857301: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:483] Coordination agent has successfully shut down. 2023-03-10 20:20:28.857468: I tensorflow/compiler/xla/pjrt/distributed/client.cc:518] Distributed task shutdown result: OK 2023-03-10 20:20:28.857516: I tensorflow/compiler/xla/pjrt/distributed/client.cc:518] Distributed task shutdown result: OK [ OK ] ClientServerTests/ClientServerTest.ConnectAndShutdownAreBarriers/CoordinationService (125 ms) [ RUN ] ClientServerTests/ClientServerTest.ConnectAndShutdownAreBarriers/DistributedRuntimeService 2023-03-10 20:20:28.896889: I tensorflow/compiler/xla/pjrt/distributed/client.cc:244] Connected to distributed JAX controller 2023-03-10 20:20:28.896941: I tensorflow/compiler/xla/pjrt/distributed/client.cc:278] Waiting for all distributed JAX tasks to shut down. 2023-03-10 20:20:28.897552: I tensorflow/compiler/xla/pjrt/distributed/client.cc:244] Connected to distributed JAX controller 2023-03-10 20:20:28.906434: I tensorflow/compiler/xla/pjrt/distributed/client.cc:244] Connected to distributed JAX controller 2023-03-10 20:20:28.906496: I tensorflow/compiler/xla/pjrt/distributed/client.cc:278] Waiting for all distributed JAX tasks to shut down. 2023-03-10 20:20:28.910903: I tensorflow/compiler/xla/pjrt/distributed/client.cc:278] Waiting for all distributed JAX tasks to shut down. 2023-03-10 20:20:28.911367: I tensorflow/compiler/xla/pjrt/distributed/client.cc:296] Distributed task shutdown result: OK 2023-03-10 20:20:28.911422: I tensorflow/compiler/xla/pjrt/distributed/client.cc:296] Distributed task shutdown result: OK 2023-03-10 20:20:28.915144: I tensorflow/compiler/xla/pjrt/distributed/client.cc:296] Distributed task shutdown result: OK [ OK ] ClientServerTests/ClientServerTest.ConnectAndShutdownAreBarriers/DistributedRuntimeService (51 ms) [ RUN ] ClientServerTests/ClientServerTest.ConnectAndEnumerateDevices/CoordinationService 2023-03-10 20:20:28.918473: I tensorflow/compiler/xla/pjrt/distributed/service.cc:522] Experimental coordination service is enabled. 2023-03-10 20:20:28.919279: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:1 has connected to coordination service. Incarnation: 15469061490964801938 2023-03-10 20:20:28.955670: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:0 has connected to coordination service. Incarnation: 4302604639411784123 2023-03-10 20:20:28.955891: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:28.956465: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:28.956841: I tensorflow/compiler/xla/pjrt/distributed/client.cc:508] Connected to distributed JAX controller 2023-03-10 20:20:28.956915: I tensorflow/compiler/xla/pjrt/distributed/client.cc:508] Connected to distributed JAX controller 2023-03-10 20:20:29.977101: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:464] Coordination agent has initiated Shutdown(). 2023-03-10 20:20:29.977153: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:464] Coordination agent has initiated Shutdown(). 2023-03-10 20:20:29.977311: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:1136] Shutdown barrier in coordination service has passed. 2023-03-10 20:20:29.977348: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:630] /job:jax_worker/replica:0/task:0 has disconnected from coordination service. 2023-03-10 20:20:29.977363: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:630] /job:jax_worker/replica:0/task:1 has disconnected from coordination service. 2023-03-10 20:20:29.977499: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:483] Coordination agent has successfully shut down. 2023-03-10 20:20:29.977557: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:483] Coordination agent has successfully shut down. [ OK ] ClientServerTests/ClientServerTest.ConnectAndEnumerateDevices/CoordinationService (1070 ms) [ RUN ] ClientServerTests/ClientServerTest.ConnectAndEnumerateDevices/DistributedRuntimeService 2023-03-10 20:20:30.020683: I tensorflow/compiler/xla/pjrt/distributed/client.cc:244] Connected to distributed JAX controller 2023-03-10 20:20:30.026656: I tensorflow/compiler/xla/pjrt/distributed/client.cc:244] Connected to distributed JAX controller 2023-03-10 20:20:31.047048: I tensorflow/compiler/xla/pjrt/distributed/client.cc:278] Waiting for all distributed JAX tasks to shut down. 2023-03-10 20:20:31.099601: I tensorflow/compiler/xla/pjrt/distributed/client.cc:278] Waiting for all distributed JAX tasks to shut down. 2023-03-10 20:20:31.100400: I tensorflow/compiler/xla/pjrt/distributed/client.cc:296] Distributed task shutdown result: OK 2023-03-10 20:20:31.100527: I tensorflow/compiler/xla/pjrt/distributed/client.cc:296] Distributed task shutdown result: OK [ OK ] ClientServerTests/ClientServerTest.ConnectAndEnumerateDevices/DistributedRuntimeService (1128 ms) [ RUN ] ClientServerTests/ClientServerTest.EnumerateElevenDevices/CoordinationService 2023-03-10 20:20:31.117212: I tensorflow/compiler/xla/pjrt/distributed/service.cc:522] Experimental coordination service is enabled. 2023-03-10 20:20:31.136565: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:2 has connected to coordination service. Incarnation: 9598557727093569398 2023-03-10 20:20:31.136676: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:3 has connected to coordination service. Incarnation: 1295002159742489584 2023-03-10 20:20:31.136717: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:7 has connected to coordination service. Incarnation: 1172284200623303055 2023-03-10 20:20:31.136760: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:1 has connected to coordination service. Incarnation: 11144970743079400227 2023-03-10 20:20:31.136794: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:8 has connected to coordination service. Incarnation: 13382098123436968366 2023-03-10 20:20:31.136836: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:4 has connected to coordination service. Incarnation: 6586216145458358656 2023-03-10 20:20:31.137041: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:31.137279: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:31.137501: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:31.143685: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:6 has connected to coordination service. Incarnation: 6571115489899840334 2023-03-10 20:20:31.143875: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:31.146478: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:5 has connected to coordination service. Incarnation: 758796936014376289 2023-03-10 20:20:31.146625: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:31.147057: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:0 has connected to coordination service. Incarnation: 5782434246340137378 2023-03-10 20:20:31.147215: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:31.150233: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:31.150611: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:31.196197: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:10 has connected to coordination service. Incarnation: 11544477834916777694 2023-03-10 20:20:31.196958: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:9 has connected to coordination service. Incarnation: 7796208467643186539 2023-03-10 20:20:31.197038: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:31.206177: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:31.216628: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:31.236528: I tensorflow/compiler/xla/pjrt/distributed/client.cc:508] Connected to distributed JAX controller 2023-03-10 20:20:31.236647: I tensorflow/compiler/xla/pjrt/distributed/client.cc:508] Connected to distributed JAX controller 2023-03-10 20:20:31.236700: I tensorflow/compiler/xla/pjrt/distributed/client.cc:508] Connected to distributed JAX controller 2023-03-10 20:20:31.236755: I tensorflow/compiler/xla/pjrt/distributed/client.cc:508] Connected to distributed JAX controller 2023-03-10 20:20:31.236812: I tensorflow/compiler/xla/pjrt/distributed/client.cc:508] Connected to distributed JAX controller 2023-03-10 20:20:31.236830: I tensorflow/compiler/xla/pjrt/distributed/client.cc:508] Connected to distributed JAX controller 2023-03-10 20:20:31.237020: I tensorflow/compiler/xla/pjrt/distributed/client.cc:508] Connected to distributed JAX controller 2023-03-10 20:20:31.246178: I tensorflow/compiler/xla/pjrt/distributed/client.cc:508] Connected to distributed JAX controller 2023-03-10 20:20:31.246440: I tensorflow/compiler/xla/pjrt/distributed/client.cc:508] Connected to distributed JAX controller 2023-03-10 20:20:31.256226: I tensorflow/compiler/xla/pjrt/distributed/client.cc:508] Connected to distributed JAX controller 2023-03-10 20:20:31.256295: I tensorflow/compiler/xla/pjrt/distributed/client.cc:508] Connected to distributed JAX controller 2023-03-10 20:20:31.256835: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:464] Coordination agent has initiated Shutdown(). 2023-03-10 20:20:31.257029: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:464] Coordination agent has initiated Shutdown(). 2023-03-10 20:20:31.258317: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:464] Coordination agent has initiated Shutdown(). 2023-03-10 20:20:31.258350: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:464] Coordination agent has initiated Shutdown(). 2023-03-10 20:20:31.259629: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:464] Coordination agent has initiated Shutdown(). 2023-03-10 20:20:31.259673: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:464] Coordination agent has initiated Shutdown(). 2023-03-10 20:20:31.263130: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:464] Coordination agent has initiated Shutdown(). 2023-03-10 20:20:31.276454: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:464] Coordination agent has initiated Shutdown(). 2023-03-10 20:20:31.276675: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:464] Coordination agent has initiated Shutdown(). 2023-03-10 20:20:31.276786: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:464] Coordination agent has initiated Shutdown(). 2023-03-10 20:20:31.276889: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:464] Coordination agent has initiated Shutdown(). 2023-03-10 20:20:31.277122: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:1136] Shutdown barrier in coordination service has passed. 2023-03-10 20:20:31.277166: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:630] /job:jax_worker/replica:0/task:6 has disconnected from coordination service. 2023-03-10 20:20:31.277180: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:630] /job:jax_worker/replica:0/task:0 has disconnected from coordination service. 2023-03-10 20:20:31.277192: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:630] /job:jax_worker/replica:0/task:3 has disconnected from coordination service. 2023-03-10 20:20:31.277204: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:630] /job:jax_worker/replica:0/task:4 has disconnected from coordination service. 2023-03-10 20:20:31.277215: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:630] /job:jax_worker/replica:0/task:1 has disconnected from coordination service. 2023-03-10 20:20:31.277226: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:630] /job:jax_worker/replica:0/task:10 has disconnected from coordination service. 2023-03-10 20:20:31.277237: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:630] /job:jax_worker/replica:0/task:8 has disconnected from coordination service. 2023-03-10 20:20:31.277248: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:630] /job:jax_worker/replica:0/task:2 has disconnected from coordination service. 2023-03-10 20:20:31.277259: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:630] /job:jax_worker/replica:0/task:5 has disconnected from coordination service. 2023-03-10 20:20:31.277270: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:630] /job:jax_worker/replica:0/task:7 has disconnected from coordination service. 2023-03-10 20:20:31.277281: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:630] /job:jax_worker/replica:0/task:9 has disconnected from coordination service. 2023-03-10 20:20:31.277436: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:483] Coordination agent has successfully shut down. 2023-03-10 20:20:31.277472: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:483] Coordination agent has successfully shut down. 2023-03-10 20:20:31.277783: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:483] Coordination agent has successfully shut down. 2023-03-10 20:20:31.277819: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:483] Coordination agent has successfully shut down. 2023-03-10 20:20:31.277967: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:483] Coordination agent has successfully shut down. 2023-03-10 20:20:31.278003: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:483] Coordination agent has successfully shut down. 2023-03-10 20:20:31.278032: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:483] Coordination agent has successfully shut down. 2023-03-10 20:20:31.278306: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:483] Coordination agent has successfully shut down. 2023-03-10 20:20:31.278419: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:483] Coordination agent has successfully shut down. 2023-03-10 20:20:31.278521: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:483] Coordination agent has successfully shut down. 2023-03-10 20:20:31.278693: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:483] Coordination agent has successfully shut down. [ OK ] ClientServerTests/ClientServerTest.EnumerateElevenDevices/CoordinationService (201 ms) [ RUN ] ClientServerTests/ClientServerTest.EnumerateElevenDevices/DistributedRuntimeService 2023-03-10 20:20:31.416748: I tensorflow/compiler/xla/pjrt/distributed/client.cc:244] Connected to distributed JAX controller 2023-03-10 20:20:31.417056: I tensorflow/compiler/xla/pjrt/distributed/client.cc:244] Connected to distributed JAX controller 2023-03-10 20:20:31.417418: I tensorflow/compiler/xla/pjrt/distributed/client.cc:244] Connected to distributed JAX controller 2023-03-10 20:20:31.417648: I tensorflow/compiler/xla/pjrt/distributed/client.cc:244] Connected to distributed JAX controller 2023-03-10 20:20:31.417878: I tensorflow/compiler/xla/pjrt/distributed/client.cc:244] Connected to distributed JAX controller 2023-03-10 20:20:31.417956: I tensorflow/compiler/xla/pjrt/distributed/client.cc:244] Connected to distributed JAX controller 2023-03-10 20:20:31.426409: I tensorflow/compiler/xla/pjrt/distributed/client.cc:244] Connected to distributed JAX controller 2023-03-10 20:20:31.426482: I tensorflow/compiler/xla/pjrt/distributed/client.cc:244] Connected to distributed JAX controller 2023-03-10 20:20:31.436937: I tensorflow/compiler/xla/pjrt/distributed/client.cc:244] Connected to distributed JAX controller 2023-03-10 20:20:31.446378: I tensorflow/compiler/xla/pjrt/distributed/client.cc:244] Connected to distributed JAX controller 2023-03-10 20:20:31.450458: I tensorflow/compiler/xla/pjrt/distributed/client.cc:244] Connected to distributed JAX controller 2023-03-10 20:20:31.466829: I tensorflow/compiler/xla/pjrt/distributed/client.cc:278] Waiting for all distributed JAX tasks to shut down. 2023-03-10 20:20:31.466893: I tensorflow/compiler/xla/pjrt/distributed/client.cc:278] Waiting for all distributed JAX tasks to shut down. 2023-03-10 20:20:31.466931: I tensorflow/compiler/xla/pjrt/distributed/client.cc:278] Waiting for all distributed JAX tasks to shut down. 2023-03-10 20:20:31.467220: I tensorflow/compiler/xla/pjrt/distributed/client.cc:278] Waiting for all distributed JAX tasks to shut down. 2023-03-10 20:20:31.467390: I tensorflow/compiler/xla/pjrt/distributed/client.cc:278] Waiting for all distributed JAX tasks to shut down. 2023-03-10 20:20:31.467850: I tensorflow/compiler/xla/pjrt/distributed/client.cc:278] Waiting for all distributed JAX tasks to shut down. 2023-03-10 20:20:31.476285: I tensorflow/compiler/xla/pjrt/distributed/client.cc:278] Waiting for all distributed JAX tasks to shut down. 2023-03-10 20:20:31.476347: I tensorflow/compiler/xla/pjrt/distributed/client.cc:278] Waiting for all distributed JAX tasks to shut down. 2023-03-10 20:20:31.476421: I tensorflow/compiler/xla/pjrt/distributed/client.cc:278] Waiting for all distributed JAX tasks to shut down. 2023-03-10 20:20:31.476550: I tensorflow/compiler/xla/pjrt/distributed/client.cc:278] Waiting for all distributed JAX tasks to shut down. 2023-03-10 20:20:31.486449: I tensorflow/compiler/xla/pjrt/distributed/client.cc:278] Waiting for all distributed JAX tasks to shut down. 2023-03-10 20:20:31.496483: I tensorflow/compiler/xla/pjrt/distributed/client.cc:296] Distributed task shutdown result: OK 2023-03-10 20:20:31.496601: I tensorflow/compiler/xla/pjrt/distributed/client.cc:296] Distributed task shutdown result: OK 2023-03-10 20:20:31.496641: I tensorflow/compiler/xla/pjrt/distributed/client.cc:296] Distributed task shutdown result: OK 2023-03-10 20:20:31.496716: I tensorflow/compiler/xla/pjrt/distributed/client.cc:296] Distributed task shutdown result: OK 2023-03-10 20:20:31.496846: I tensorflow/compiler/xla/pjrt/distributed/client.cc:296] Distributed task shutdown result: OK 2023-03-10 20:20:31.496955: I tensorflow/compiler/xla/pjrt/distributed/client.cc:296] Distributed task shutdown result: OK 2023-03-10 20:20:31.497013: I tensorflow/compiler/xla/pjrt/distributed/client.cc:296] Distributed task shutdown result: OK 2023-03-10 20:20:31.499067: I tensorflow/compiler/xla/pjrt/distributed/client.cc:296] Distributed task shutdown result: OK 2023-03-10 20:20:31.503145: I tensorflow/compiler/xla/pjrt/distributed/client.cc:296] Distributed task shutdown result: OK 2023-03-10 20:20:31.503242: I tensorflow/compiler/xla/pjrt/distributed/client.cc:296] Distributed task shutdown result: OK 2023-03-10 20:20:31.506205: I tensorflow/compiler/xla/pjrt/distributed/client.cc:296] Distributed task shutdown result: OK [ OK ] ClientServerTests/ClientServerTest.EnumerateElevenDevices/DistributedRuntimeService (208 ms) [ RUN ] ClientServerTests/ClientServerTest.ZeroInitTimeoutShouldStillWaitForOtherTasks/CoordinationService 2023-03-10 20:20:31.546727: I tensorflow/compiler/xla/pjrt/distributed/service.cc:522] Experimental coordination service is enabled. 2023-03-10 20:20:31.566213: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:0 has connected to coordination service. Incarnation: 9079344095076790944 2023-03-10 20:20:31.576256: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:36.566310: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:525] /job:jax_worker/replica:0/task:1 has connected to coordination service. Incarnation: 2742233341858543835 2023-03-10 20:20:39.024760: E tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:737] Coordination agent is in ERROR: DEADLINE_EXCEEDED: Deadline Exceeded Additional GRPC error information from remote target unknown_target_for_coordination_leader: :{"created":"@1678479637.686597401","description":"Deadline Exceeded","file":"external/com_github_grpc_grpc/src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4} 2023-03-10 20:20:39.026016: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. 2023-03-10 20:20:39.026053: E tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:961] /job:jax_worker/replica:0/task:1 has been set to ERROR in coordination service: UNAVAILABLE: Task /job:jax_worker/replica:0/task:1 heartbeat timeout. This indicates that the remote task has failed, got preempted, or crashed unexpectedly. [type.googleapis.com/tensorflow.CoordinationServiceError=''] 2023-03-10 20:20:39.026100: E tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:411] Stopping coordination service as heartbeat has timed out for /job:jax_worker/replica:0/task:1 and there is no service-to-client connection 2023-03-10 20:20:39.026159: I tensorflow/compiler/xla/pjrt/distributed/client.cc:510] Failed to connect to distributed JAX controller: INTERNAL: Barrier failed from a task error. Barrier Id: PjRT_Client_Connect, Task: /job:jax_worker/replica:0/task:1 Additional GRPC error information from remote target unknown_target_for_coordination_leader: :{"created":"@1678479639.025009115","description":"Error received from peer inproc","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Barrier failed from a task error. Barrier Id: PjRT_Client_Connect, Task: /job:jax_worker/replica:0/task:1","grpc_status":13} [type.googleapis.com/tensorflow.CoordinationServiceError=''] 2023-03-10 20:20:39.026413: E tensorflow/compiler/xla/pjrt/distributed/client.cc:472] Coordination service agent in error status: DEADLINE_EXCEEDED: Deadline Exceeded Additional GRPC error information from remote target unknown_target_for_coordination_leader: :{"created":"@1678479637.686597401","description":"Deadline Exceeded","file":"external/com_github_grpc_grpc/src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4} 2023-03-10 20:20:39.026478: F ./tensorflow/compiler/xla/pjrt/distributed/client.h:77] Terminating process because the coordinator detected missing heartbeats. This most likely indicates that another task died; see the other task logs for more details. Status: DEADLINE_EXCEEDED: Deadline Exceeded Additional GRPC error information from remote target unknown_target_for_coordination_leader: :{"created":"@1678479637.686597401","description":"Deadline Exceeded","file":"external/com_github_grpc_grpc/src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4} *** Received signal 6 *** *** BEGIN MANGLED STACK TRACE *** 2023-03-10 20:20:39.026724: I tensorflow/compiler/xla/pjrt/distributed/client.cc:510] Failed to connect to distributed JAX controller: INVALID_ARGUMENT: A non-participating task (/job:jax_worker/replica:0/task:1) called the barrier: PjRT_Client_Connect Additional GRPC error information from remote target unknown_target_for_coordination_leader: :{"created":"@1678479639.026450150","description":"Error received from peer inproc","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"A non-participating task (/job:jax_worker/replica:0/task:1) called the barrier: PjRT_Client_Connect","grpc_status":3} [type.googleapis.com/tensorflow.CoordinationServiceError=''] 2023-03-10 20:20:39.026754: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:464] Coordination agent has initiated Shutdown(). 2023-03-10 20:20:39.026782: E tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:737] Coordination agent is in ERROR: INVALID_ARGUMENT: Unexpected task request with task_name=/job:jax_worker/replica:0/task:1 Additional GRPC error information from remote target unknown_target_for_coordination_leader: :{"created":"@1678479639.026561341","description":"Error received from peer inproc","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Unexpected task request with task_name=/job:jax_worker/replica:0/task:1","grpc_status":3} [type.googleapis.com/tensorflow.CoordinationServiceError=''] 2023-03-10 20:20:39.026818: E tensorflow/compiler/xla/pjrt/distributed/client.cc:472] Coordination service agent in error status: INVALID_ARGUMENT: Unexpected task request with task_name=/job:jax_worker/replica:0/task:1 Additional GRPC error information from remote target unknown_target_for_coordination_leader: :{"created":"@1678479639.026561341","description":"Error received from peer inproc","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Unexpected task request with task_name=/job:jax_worker/replica:0/task:1","grpc_status":3} [type.googleapis.com/tensorflow.CoordinationServiceError=''] 2023-03-10 20:20:39.026837: F ./tensorflow/compiler/xla/pjrt/distributed/client.h:77] Terminating process because the coordinator detected missing heartbeats. This most likely indicates that another task died; see the other task logs for more details. Status: INVALID_ARGUMENT: Unexpected task request with task_name=/job:jax_worker/replica:0/task:1 Additional GRPC error information from remote target unknown_target_for_coordination_leader: :{"created":"@1678479639.026561341","description":"Error received from peer inproc","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Unexpected task request with task_name=/job:jax_worker/replica:0/task:1","grpc_status":3} [type.googleapis.com/tensorflow.CoordinationServiceError=''] ================================================================================ ==================== Test output for //tensorflow/python/kernel_tests/proto:decode_proto_op_test: Running tests under Python 3.9.16: /usr/local/bin/python3 [ RUN ] DecodeProtoOpTest.testBinaryDisordereddefaults INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisordereddefaults): 0.09s I0310 20:22:50.108673 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisordereddefaults): 0.09s [ FAILED ] DecodeProtoOpTest.testBinaryDisordereddefaults [ RUN ] DecodeProtoOpTest.testBinaryDisorderedextension INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedextension): 0.01s I0310 20:22:50.116296 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedextension): 0.01s [ OK ] DecodeProtoOpTest.testBinaryDisorderedextension [ RUN ] DecodeProtoOpTest.testBinaryDisorderedminmax INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedminmax): 0.01s I0310 20:22:50.127372 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedminmax): 0.01s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedminmax [ RUN ] DecodeProtoOpTest.testBinaryDisorderednested INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderednested): 0.0s I0310 20:22:50.132464 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderednested): 0.0s [ OK ] DecodeProtoOpTest.testBinaryDisorderednested [ RUN ] DecodeProtoOpTest.testBinaryDisorderedoptional INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedoptional): 0.0s I0310 20:22:50.137005 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedoptional): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedoptional [ RUN ] DecodeProtoOpTest.testBinaryDisorderedpromote INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedpromote): 0.0s I0310 20:22:50.142480 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedpromote): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedpromote [ RUN ] DecodeProtoOpTest.testBinaryDisorderedragged INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedragged): 0.0s I0310 20:22:50.147242 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedragged): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedragged [ RUN ] DecodeProtoOpTest.testBinaryDisorderedshaped_batch INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedshaped_batch): 0.0s I0310 20:22:50.151988 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedshaped_batch): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedshaped_batch [ RUN ] DecodeProtoOpTest.testBinaryDisorderedsimple INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedsimple): 0.0s I0310 20:22:50.156836 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedsimple): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedsimple [ RUN ] DecodeProtoOpTest.testBinarydefaults INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinarydefaults): 0.01s I0310 20:22:50.163226 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinarydefaults): 0.01s [ FAILED ] DecodeProtoOpTest.testBinarydefaults [ RUN ] DecodeProtoOpTest.testBinaryextension INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryextension): 0.0s I0310 20:22:50.168512 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryextension): 0.0s [ OK ] DecodeProtoOpTest.testBinaryextension [ RUN ] DecodeProtoOpTest.testBinaryminmax INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryminmax): 0.0s I0310 20:22:50.174285 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryminmax): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryminmax [ RUN ] DecodeProtoOpTest.testBinarynested INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinarynested): 0.0s I0310 20:22:50.179674 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinarynested): 0.0s [ OK ] DecodeProtoOpTest.testBinarynested [ RUN ] DecodeProtoOpTest.testBinaryoptional INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryoptional): 0.01s I0310 20:22:50.185674 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryoptional): 0.01s [ FAILED ] DecodeProtoOpTest.testBinaryoptional [ RUN ] DecodeProtoOpTest.testBinarypromote INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinarypromote): 0.0s I0310 20:22:50.191858 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinarypromote): 0.0s [ FAILED ] DecodeProtoOpTest.testBinarypromote [ RUN ] DecodeProtoOpTest.testBinaryragged INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryragged): 0.0s I0310 20:22:50.198009 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryragged): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryragged [ RUN ] DecodeProtoOpTest.testBinaryshaped_batch INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryshaped_batch): 0.0s I0310 20:22:50.203693 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryshaped_batch): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryshaped_batch [ RUN ] DecodeProtoOpTest.testBinarysimple INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinarysimple): 0.0s I0310 20:22:50.209333 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinarysimple): 0.0s [ FAILED ] DecodeProtoOpTest.testBinarysimple [ RUN ] DecodeProtoOpTest.testCorruptProtobuf0 (False) 2023-03-10 20:22:50.214725: W tensorflow/core/framework/op_kernel.cc:1829] OP_REQUIRES failed at decode_proto_op.cc:851 : DATA_LOSS: CountFields: Failed to consume entire buffer INFO:tensorflow:time(__main__.DecodeProtoOpTest.testCorruptProtobuf0 (False)): 0.0s I0310 20:22:50.215039 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testCorruptProtobuf0 (False)): 0.0s [ OK ] DecodeProtoOpTest.testCorruptProtobuf0 (False) [ RUN ] DecodeProtoOpTest.testCorruptProtobuf1 (True) INFO:tensorflow:time(__main__.DecodeProtoOpTest.testCorruptProtobuf1 (True)): 0.0s I0310 20:22:50.219329 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testCorruptProtobuf1 (True)): 0.0s [ OK ] DecodeProtoOpTest.testCorruptProtobuf1 (True) [ RUN ] DecodeProtoOpTest.testOutOfOrderRepeated INFO:tensorflow:time(__main__.DecodeProtoOpTest.testOutOfOrderRepeated): 0.02s I0310 20:22:50.242303 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testOutOfOrderRepeated): 0.02s [ OK ] DecodeProtoOpTest.testOutOfOrderRepeated [ RUN ] DecodeProtoOpTest.testPackeddefaults INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackeddefaults): 0.01s I0310 20:22:50.249263 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackeddefaults): 0.01s [ FAILED ] DecodeProtoOpTest.testPackeddefaults [ RUN ] DecodeProtoOpTest.testPackedminmax INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedminmax): 0.01s I0310 20:22:50.257975 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedminmax): 0.01s [ FAILED ] DecodeProtoOpTest.testPackedminmax [ RUN ] DecodeProtoOpTest.testPackednested INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackednested): 0.0s I0310 20:22:50.264179 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackednested): 0.0s [ OK ] DecodeProtoOpTest.testPackednested [ RUN ] DecodeProtoOpTest.testPackedoptional INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedoptional): 0.0s I0310 20:22:50.269310 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedoptional): 0.0s [ FAILED ] DecodeProtoOpTest.testPackedoptional [ RUN ] DecodeProtoOpTest.testPackedpromote INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedpromote): 0.01s I0310 20:22:50.275706 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedpromote): 0.01s [ FAILED ] DecodeProtoOpTest.testPackedpromote [ RUN ] DecodeProtoOpTest.testPackedragged INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedragged): 0.0s I0310 20:22:50.282049 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedragged): 0.0s [ FAILED ] DecodeProtoOpTest.testPackedragged [ RUN ] DecodeProtoOpTest.testPackedshaped_batch INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedshaped_batch): 0.01s I0310 20:22:50.288778 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedshaped_batch): 0.01s [ FAILED ] DecodeProtoOpTest.testPackedshaped_batch [ RUN ] DecodeProtoOpTest.testPackedsimple INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedsimple): 0.01s I0310 20:22:50.295403 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedsimple): 0.01s [ FAILED ] DecodeProtoOpTest.testPackedsimple [ RUN ] DecodeProtoOpTest.testSanitizerGooddefaults INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGooddefaults): 0.01s I0310 20:22:50.302965 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGooddefaults): 0.01s [ FAILED ] DecodeProtoOpTest.testSanitizerGooddefaults [ RUN ] DecodeProtoOpTest.testSanitizerGoodextension INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodextension): 0.0s I0310 20:22:50.308762 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodextension): 0.0s [ OK ] DecodeProtoOpTest.testSanitizerGoodextension [ RUN ] DecodeProtoOpTest.testSanitizerGoodminmax INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodminmax): 0.01s I0310 20:22:50.315271 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodminmax): 0.01s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodminmax [ RUN ] DecodeProtoOpTest.testSanitizerGoodnested INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodnested): 0.0s I0310 20:22:50.321207 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodnested): 0.0s [ OK ] DecodeProtoOpTest.testSanitizerGoodnested [ RUN ] DecodeProtoOpTest.testSanitizerGoodoptional INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodoptional): 0.0s I0310 20:22:50.326140 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodoptional): 0.0s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodoptional [ RUN ] DecodeProtoOpTest.testSanitizerGoodpromote INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodpromote): 0.0s I0310 20:22:50.331851 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodpromote): 0.0s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodpromote [ RUN ] DecodeProtoOpTest.testSanitizerGoodragged INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodragged): 0.0s I0310 20:22:50.337061 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodragged): 0.0s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodragged [ RUN ] DecodeProtoOpTest.testSanitizerGoodshaped_batch INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodshaped_batch): 0.0s I0310 20:22:50.342236 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodshaped_batch): 0.0s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodshaped_batch [ RUN ] DecodeProtoOpTest.testSanitizerGoodsimple INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodsimple): 0.0s I0310 20:22:50.347598 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodsimple): 0.0s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodsimple [ RUN ] DecodeProtoOpTest.testTextdefaults INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextdefaults): 0.01s I0310 20:22:50.354222 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextdefaults): 0.01s [ FAILED ] DecodeProtoOpTest.testTextdefaults [ RUN ] DecodeProtoOpTest.testTextextension INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextextension): 0.01s I0310 20:22:50.361812 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextextension): 0.01s [ OK ] DecodeProtoOpTest.testTextextension [ RUN ] DecodeProtoOpTest.testTextminmax INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextminmax): 0.01s I0310 20:22:50.368540 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextminmax): 0.01s [ FAILED ] DecodeProtoOpTest.testTextminmax [ RUN ] DecodeProtoOpTest.testTextnested INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextnested): 0.0s I0310 20:22:50.374277 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextnested): 0.0s [ OK ] DecodeProtoOpTest.testTextnested [ RUN ] DecodeProtoOpTest.testTextoptional INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextoptional): 0.0s I0310 20:22:50.379359 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextoptional): 0.0s [ FAILED ] DecodeProtoOpTest.testTextoptional [ RUN ] DecodeProtoOpTest.testTextpromote INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextpromote): 0.0s I0310 20:22:50.385116 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextpromote): 0.0s [ FAILED ] DecodeProtoOpTest.testTextpromote [ RUN ] DecodeProtoOpTest.testTextragged INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextragged): 0.0s I0310 20:22:50.390682 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextragged): 0.0s [ FAILED ] DecodeProtoOpTest.testTextragged [ RUN ] DecodeProtoOpTest.testTextshaped_batch INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextshaped_batch): 0.0s I0310 20:22:50.396369 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextshaped_batch): 0.0s [ FAILED ] DecodeProtoOpTest.testTextshaped_batch [ RUN ] DecodeProtoOpTest.testTextsimple INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextsimple): 0.0s I0310 20:22:50.401797 281473172009856 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextsimple): 0.0s [ FAILED ] DecodeProtoOpTest.testTextsimple [ RUN ] DecodeProtoOpTest.test_session [ SKIPPED ] DecodeProtoOpTest.test_session ====================================================================== ERROR: testBinaryDisordereddefaults (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisordereddefaults testBinaryDisordereddefaults(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedminmax (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedminmax testBinaryDisorderedminmax(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedoptional (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedoptional testBinaryDisorderedoptional(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedpromote (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedpromote testBinaryDisorderedpromote(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedragged (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedragged testBinaryDisorderedragged(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedshaped_batch (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedshaped_batch testBinaryDisorderedshaped_batch(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedsimple (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedsimple testBinaryDisorderedsimple(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinarydefaults (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinarydefaults testBinarydefaults(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryminmax (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryminmax testBinaryminmax(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryoptional (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryoptional testBinaryoptional(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinarypromote (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinarypromote testBinarypromote(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryragged (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryragged testBinaryragged(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryshaped_batch (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryshaped_batch testBinaryshaped_batch(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinarysimple (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinarysimple testBinarysimple(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackeddefaults (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackeddefaults testPackeddefaults(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedminmax (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedminmax testPackedminmax(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedoptional (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedoptional testPackedoptional(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedpromote (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedpromote testPackedpromote(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedragged (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedragged testPackedragged(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedshaped_batch (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedshaped_batch testPackedshaped_batch(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedsimple (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedsimple testPackedsimple(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGooddefaults (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGooddefaults testSanitizerGooddefaults(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodminmax (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodminmax testSanitizerGoodminmax(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodoptional (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodoptional testSanitizerGoodoptional(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodpromote (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodpromote testSanitizerGoodpromote(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodragged (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodragged testSanitizerGoodragged(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodshaped_batch (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodshaped_batch testSanitizerGoodshaped_batch(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodsimple (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodsimple testSanitizerGoodsimple(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextdefaults (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextdefaults testTextdefaults(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextminmax (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextminmax testTextminmax(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextoptional (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextoptional testTextoptional(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextpromote (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextpromote testTextpromote(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextragged (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextragged testTextragged(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextshaped_batch (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextshaped_batch testTextshaped_batch(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextsimple (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextsimple testTextsimple(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ---------------------------------------------------------------------- Ran 48 tests in 0.385s FAILED (errors=35, skipped=1) ================================================================================ ==================== Test output for //tensorflow/python/kernel_tests/proto:decode_proto_op_test: Running tests under Python 3.9.16: /usr/local/bin/python3 [ RUN ] DecodeProtoOpTest.testBinaryDisordereddefaults INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisordereddefaults): 0.07s I0310 20:22:53.582231 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisordereddefaults): 0.07s [ FAILED ] DecodeProtoOpTest.testBinaryDisordereddefaults [ RUN ] DecodeProtoOpTest.testBinaryDisorderedextension INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedextension): 0.0s I0310 20:22:53.588765 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedextension): 0.0s [ OK ] DecodeProtoOpTest.testBinaryDisorderedextension [ RUN ] DecodeProtoOpTest.testBinaryDisorderedminmax INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedminmax): 0.01s I0310 20:22:53.594835 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedminmax): 0.01s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedminmax [ RUN ] DecodeProtoOpTest.testBinaryDisorderednested INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderednested): 0.0s I0310 20:22:53.600512 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderednested): 0.0s [ OK ] DecodeProtoOpTest.testBinaryDisorderednested [ RUN ] DecodeProtoOpTest.testBinaryDisorderedoptional INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedoptional): 0.0s I0310 20:22:53.605439 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedoptional): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedoptional [ RUN ] DecodeProtoOpTest.testBinaryDisorderedpromote INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedpromote): 0.0s I0310 20:22:53.611100 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedpromote): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedpromote [ RUN ] DecodeProtoOpTest.testBinaryDisorderedragged INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedragged): 0.0s I0310 20:22:53.616286 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedragged): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedragged [ RUN ] DecodeProtoOpTest.testBinaryDisorderedshaped_batch INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedshaped_batch): 0.0s I0310 20:22:53.621242 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedshaped_batch): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedshaped_batch [ RUN ] DecodeProtoOpTest.testBinaryDisorderedsimple INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedsimple): 0.0s I0310 20:22:53.626369 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedsimple): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedsimple [ RUN ] DecodeProtoOpTest.testBinarydefaults INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinarydefaults): 0.01s I0310 20:22:53.632786 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinarydefaults): 0.01s [ FAILED ] DecodeProtoOpTest.testBinarydefaults [ RUN ] DecodeProtoOpTest.testBinaryextension INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryextension): 0.0s I0310 20:22:53.638410 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryextension): 0.0s [ OK ] DecodeProtoOpTest.testBinaryextension [ RUN ] DecodeProtoOpTest.testBinaryminmax INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryminmax): 0.01s I0310 20:22:53.644516 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryminmax): 0.01s [ FAILED ] DecodeProtoOpTest.testBinaryminmax [ RUN ] DecodeProtoOpTest.testBinarynested INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinarynested): 0.0s I0310 20:22:53.650368 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinarynested): 0.0s [ OK ] DecodeProtoOpTest.testBinarynested [ RUN ] DecodeProtoOpTest.testBinaryoptional INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryoptional): 0.0s I0310 20:22:53.655228 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryoptional): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryoptional [ RUN ] DecodeProtoOpTest.testBinarypromote INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinarypromote): 0.0s I0310 20:22:53.660750 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinarypromote): 0.0s [ FAILED ] DecodeProtoOpTest.testBinarypromote [ RUN ] DecodeProtoOpTest.testBinaryragged INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryragged): 0.0s I0310 20:22:53.665844 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryragged): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryragged [ RUN ] DecodeProtoOpTest.testBinaryshaped_batch INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryshaped_batch): 0.0s I0310 20:22:53.670922 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryshaped_batch): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryshaped_batch [ RUN ] DecodeProtoOpTest.testBinarysimple INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinarysimple): 0.0s I0310 20:22:53.675741 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinarysimple): 0.0s [ FAILED ] DecodeProtoOpTest.testBinarysimple [ RUN ] DecodeProtoOpTest.testCorruptProtobuf0 (False) 2023-03-10 20:22:53.758348: W tensorflow/core/framework/op_kernel.cc:1829] OP_REQUIRES failed at decode_proto_op.cc:851 : DATA_LOSS: CountFields: Failed to consume entire buffer INFO:tensorflow:time(__main__.DecodeProtoOpTest.testCorruptProtobuf0 (False)): 0.08s I0310 20:22:53.758691 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testCorruptProtobuf0 (False)): 0.08s [ OK ] DecodeProtoOpTest.testCorruptProtobuf0 (False) [ RUN ] DecodeProtoOpTest.testCorruptProtobuf1 (True) INFO:tensorflow:time(__main__.DecodeProtoOpTest.testCorruptProtobuf1 (True)): 0.0s I0310 20:22:53.762816 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testCorruptProtobuf1 (True)): 0.0s [ OK ] DecodeProtoOpTest.testCorruptProtobuf1 (True) [ RUN ] DecodeProtoOpTest.testOutOfOrderRepeated INFO:tensorflow:time(__main__.DecodeProtoOpTest.testOutOfOrderRepeated): 0.02s I0310 20:22:53.783834 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testOutOfOrderRepeated): 0.02s [ OK ] DecodeProtoOpTest.testOutOfOrderRepeated [ RUN ] DecodeProtoOpTest.testPackeddefaults INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackeddefaults): 0.12s I0310 20:22:53.905764 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackeddefaults): 0.12s [ FAILED ] DecodeProtoOpTest.testPackeddefaults [ RUN ] DecodeProtoOpTest.testPackedminmax INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedminmax): 0.01s I0310 20:22:53.914261 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedminmax): 0.01s [ FAILED ] DecodeProtoOpTest.testPackedminmax [ RUN ] DecodeProtoOpTest.testPackednested INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackednested): 0.0s I0310 20:22:53.919491 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackednested): 0.0s [ OK ] DecodeProtoOpTest.testPackednested [ RUN ] DecodeProtoOpTest.testPackedoptional INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedoptional): 0.0s I0310 20:22:53.924040 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedoptional): 0.0s [ FAILED ] DecodeProtoOpTest.testPackedoptional [ RUN ] DecodeProtoOpTest.testPackedpromote INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedpromote): 0.0s I0310 20:22:53.929491 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedpromote): 0.0s [ FAILED ] DecodeProtoOpTest.testPackedpromote [ RUN ] DecodeProtoOpTest.testPackedragged INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedragged): 0.0s I0310 20:22:53.934622 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedragged): 0.0s [ FAILED ] DecodeProtoOpTest.testPackedragged [ RUN ] DecodeProtoOpTest.testPackedshaped_batch INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedshaped_batch): 0.0s I0310 20:22:53.940217 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedshaped_batch): 0.0s [ FAILED ] DecodeProtoOpTest.testPackedshaped_batch [ RUN ] DecodeProtoOpTest.testPackedsimple INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedsimple): 0.0s I0310 20:22:53.945197 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedsimple): 0.0s [ FAILED ] DecodeProtoOpTest.testPackedsimple [ RUN ] DecodeProtoOpTest.testSanitizerGooddefaults INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGooddefaults): 0.0s I0310 20:22:53.951234 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGooddefaults): 0.0s [ FAILED ] DecodeProtoOpTest.testSanitizerGooddefaults [ RUN ] DecodeProtoOpTest.testSanitizerGoodextension INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodextension): 0.0s I0310 20:22:53.955905 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodextension): 0.0s [ OK ] DecodeProtoOpTest.testSanitizerGoodextension [ RUN ] DecodeProtoOpTest.testSanitizerGoodminmax INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodminmax): 0.0s I0310 20:22:53.961486 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodminmax): 0.0s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodminmax [ RUN ] DecodeProtoOpTest.testSanitizerGoodnested INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodnested): 0.0s I0310 20:22:53.966149 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodnested): 0.0s [ OK ] DecodeProtoOpTest.testSanitizerGoodnested [ RUN ] DecodeProtoOpTest.testSanitizerGoodoptional INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodoptional): 0.0s I0310 20:22:53.970333 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodoptional): 0.0s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodoptional [ RUN ] DecodeProtoOpTest.testSanitizerGoodpromote INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodpromote): 0.0s I0310 20:22:53.975190 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodpromote): 0.0s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodpromote [ RUN ] DecodeProtoOpTest.testSanitizerGoodragged INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodragged): 0.0s I0310 20:22:53.979900 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodragged): 0.0s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodragged [ RUN ] DecodeProtoOpTest.testSanitizerGoodshaped_batch INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodshaped_batch): 0.0s I0310 20:22:53.986179 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodshaped_batch): 0.0s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodshaped_batch [ RUN ] DecodeProtoOpTest.testSanitizerGoodsimple INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodsimple): 0.0s I0310 20:22:53.992216 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodsimple): 0.0s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodsimple [ RUN ] DecodeProtoOpTest.testTextdefaults INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextdefaults): 0.01s I0310 20:22:54.001563 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextdefaults): 0.01s [ FAILED ] DecodeProtoOpTest.testTextdefaults [ RUN ] DecodeProtoOpTest.testTextextension INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextextension): 0.01s I0310 20:22:54.009725 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextextension): 0.01s [ OK ] DecodeProtoOpTest.testTextextension [ RUN ] DecodeProtoOpTest.testTextminmax INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextminmax): 0.01s I0310 20:22:54.016104 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextminmax): 0.01s [ FAILED ] DecodeProtoOpTest.testTextminmax [ RUN ] DecodeProtoOpTest.testTextnested INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextnested): 0.0s I0310 20:22:54.021226 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextnested): 0.0s [ OK ] DecodeProtoOpTest.testTextnested [ RUN ] DecodeProtoOpTest.testTextoptional INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextoptional): 0.0s I0310 20:22:54.025660 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextoptional): 0.0s [ FAILED ] DecodeProtoOpTest.testTextoptional [ RUN ] DecodeProtoOpTest.testTextpromote INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextpromote): 0.0s I0310 20:22:54.030872 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextpromote): 0.0s [ FAILED ] DecodeProtoOpTest.testTextpromote [ RUN ] DecodeProtoOpTest.testTextragged INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextragged): 0.0s I0310 20:22:54.036056 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextragged): 0.0s [ FAILED ] DecodeProtoOpTest.testTextragged [ RUN ] DecodeProtoOpTest.testTextshaped_batch INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextshaped_batch): 0.0s I0310 20:22:54.041682 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextshaped_batch): 0.0s [ FAILED ] DecodeProtoOpTest.testTextshaped_batch [ RUN ] DecodeProtoOpTest.testTextsimple INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextsimple): 0.0s I0310 20:22:54.047045 281473130525568 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextsimple): 0.0s [ FAILED ] DecodeProtoOpTest.testTextsimple [ RUN ] DecodeProtoOpTest.test_session [ SKIPPED ] DecodeProtoOpTest.test_session ====================================================================== ERROR: testBinaryDisordereddefaults (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisordereddefaults testBinaryDisordereddefaults(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedminmax (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedminmax testBinaryDisorderedminmax(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedoptional (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedoptional testBinaryDisorderedoptional(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedpromote (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedpromote testBinaryDisorderedpromote(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedragged (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedragged testBinaryDisorderedragged(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedshaped_batch (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedshaped_batch testBinaryDisorderedshaped_batch(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedsimple (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedsimple testBinaryDisorderedsimple(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinarydefaults (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinarydefaults testBinarydefaults(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryminmax (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryminmax testBinaryminmax(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryoptional (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryoptional testBinaryoptional(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinarypromote (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinarypromote testBinarypromote(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryragged (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryragged testBinaryragged(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryshaped_batch (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryshaped_batch testBinaryshaped_batch(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinarysimple (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinarysimple testBinarysimple(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackeddefaults (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackeddefaults testPackeddefaults(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedminmax (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedminmax testPackedminmax(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedoptional (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedoptional testPackedoptional(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedpromote (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedpromote testPackedpromote(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedragged (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedragged testPackedragged(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedshaped_batch (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedshaped_batch testPackedshaped_batch(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedsimple (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedsimple testPackedsimple(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGooddefaults (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGooddefaults testSanitizerGooddefaults(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodminmax (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodminmax testSanitizerGoodminmax(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodoptional (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodoptional testSanitizerGoodoptional(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodpromote (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodpromote testSanitizerGoodpromote(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodragged (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodragged testSanitizerGoodragged(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodshaped_batch (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodshaped_batch testSanitizerGoodshaped_batch(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodsimple (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodsimple testSanitizerGoodsimple(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextdefaults (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextdefaults testTextdefaults(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextminmax (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextminmax testTextminmax(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextoptional (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextoptional testTextoptional(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextpromote (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextpromote testTextpromote(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextragged (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextragged testTextragged(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextshaped_batch (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextshaped_batch testTextshaped_batch(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextsimple (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextsimple testTextsimple(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ---------------------------------------------------------------------- Ran 48 tests in 0.534s FAILED (errors=35, skipped=1) ================================================================================ ==================== Test output for //tensorflow/python/kernel_tests/proto:decode_proto_op_test: Running tests under Python 3.9.16: /usr/local/bin/python3 [ RUN ] DecodeProtoOpTest.testBinaryDisordereddefaults INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisordereddefaults): 0.07s I0310 20:22:57.364198 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisordereddefaults): 0.07s [ FAILED ] DecodeProtoOpTest.testBinaryDisordereddefaults [ RUN ] DecodeProtoOpTest.testBinaryDisorderedextension INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedextension): 0.0s I0310 20:22:57.370739 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedextension): 0.0s [ OK ] DecodeProtoOpTest.testBinaryDisorderedextension [ RUN ] DecodeProtoOpTest.testBinaryDisorderedminmax INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedminmax): 0.01s I0310 20:22:57.376812 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedminmax): 0.01s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedminmax [ RUN ] DecodeProtoOpTest.testBinaryDisorderednested INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderednested): 0.0s I0310 20:22:57.381968 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderednested): 0.0s [ OK ] DecodeProtoOpTest.testBinaryDisorderednested [ RUN ] DecodeProtoOpTest.testBinaryDisorderedoptional INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedoptional): 0.0s I0310 20:22:57.386837 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedoptional): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedoptional [ RUN ] DecodeProtoOpTest.testBinaryDisorderedpromote INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedpromote): 0.0s I0310 20:22:57.392390 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedpromote): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedpromote [ RUN ] DecodeProtoOpTest.testBinaryDisorderedragged INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedragged): 0.0s I0310 20:22:57.397440 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedragged): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedragged [ RUN ] DecodeProtoOpTest.testBinaryDisorderedshaped_batch INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedshaped_batch): 0.0s I0310 20:22:57.402542 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedshaped_batch): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedshaped_batch [ RUN ] DecodeProtoOpTest.testBinaryDisorderedsimple INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryDisorderedsimple): 0.0s I0310 20:22:57.407716 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryDisorderedsimple): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryDisorderedsimple [ RUN ] DecodeProtoOpTest.testBinarydefaults INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinarydefaults): 0.01s I0310 20:22:57.414175 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinarydefaults): 0.01s [ FAILED ] DecodeProtoOpTest.testBinarydefaults [ RUN ] DecodeProtoOpTest.testBinaryextension INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryextension): 0.0s I0310 20:22:57.419541 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryextension): 0.0s [ OK ] DecodeProtoOpTest.testBinaryextension [ RUN ] DecodeProtoOpTest.testBinaryminmax INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryminmax): 0.01s I0310 20:22:57.425539 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryminmax): 0.01s [ FAILED ] DecodeProtoOpTest.testBinaryminmax [ RUN ] DecodeProtoOpTest.testBinarynested INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinarynested): 0.0s I0310 20:22:57.431038 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinarynested): 0.0s [ OK ] DecodeProtoOpTest.testBinarynested [ RUN ] DecodeProtoOpTest.testBinaryoptional INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryoptional): 0.0s I0310 20:22:57.435715 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryoptional): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryoptional [ RUN ] DecodeProtoOpTest.testBinarypromote INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinarypromote): 0.0s I0310 20:22:57.441334 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinarypromote): 0.0s [ FAILED ] DecodeProtoOpTest.testBinarypromote [ RUN ] DecodeProtoOpTest.testBinaryragged INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryragged): 0.0s I0310 20:22:57.447052 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryragged): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryragged [ RUN ] DecodeProtoOpTest.testBinaryshaped_batch INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinaryshaped_batch): 0.0s I0310 20:22:57.452591 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinaryshaped_batch): 0.0s [ FAILED ] DecodeProtoOpTest.testBinaryshaped_batch [ RUN ] DecodeProtoOpTest.testBinarysimple INFO:tensorflow:time(__main__.DecodeProtoOpTest.testBinarysimple): 0.01s I0310 20:22:57.459163 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testBinarysimple): 0.01s [ FAILED ] DecodeProtoOpTest.testBinarysimple [ RUN ] DecodeProtoOpTest.testCorruptProtobuf0 (False) 2023-03-10 20:22:57.465252: W tensorflow/core/framework/op_kernel.cc:1829] OP_REQUIRES failed at decode_proto_op.cc:851 : DATA_LOSS: CountFields: Failed to consume entire buffer INFO:tensorflow:time(__main__.DecodeProtoOpTest.testCorruptProtobuf0 (False)): 0.0s I0310 20:22:57.465682 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testCorruptProtobuf0 (False)): 0.0s [ OK ] DecodeProtoOpTest.testCorruptProtobuf0 (False) [ RUN ] DecodeProtoOpTest.testCorruptProtobuf1 (True) INFO:tensorflow:time(__main__.DecodeProtoOpTest.testCorruptProtobuf1 (True)): 0.0s I0310 20:22:57.471065 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testCorruptProtobuf1 (True)): 0.0s [ OK ] DecodeProtoOpTest.testCorruptProtobuf1 (True) [ RUN ] DecodeProtoOpTest.testOutOfOrderRepeated INFO:tensorflow:time(__main__.DecodeProtoOpTest.testOutOfOrderRepeated): 0.03s I0310 20:22:57.499302 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testOutOfOrderRepeated): 0.03s [ OK ] DecodeProtoOpTest.testOutOfOrderRepeated [ RUN ] DecodeProtoOpTest.testPackeddefaults INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackeddefaults): 0.01s I0310 20:22:57.508374 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackeddefaults): 0.01s [ FAILED ] DecodeProtoOpTest.testPackeddefaults [ RUN ] DecodeProtoOpTest.testPackedminmax INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedminmax): 0.01s I0310 20:22:57.519792 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedminmax): 0.01s [ FAILED ] DecodeProtoOpTest.testPackedminmax [ RUN ] DecodeProtoOpTest.testPackednested INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackednested): 0.01s I0310 20:22:57.527719 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackednested): 0.01s [ OK ] DecodeProtoOpTest.testPackednested [ RUN ] DecodeProtoOpTest.testPackedoptional INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedoptional): 0.01s I0310 20:22:57.535316 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedoptional): 0.01s [ FAILED ] DecodeProtoOpTest.testPackedoptional [ RUN ] DecodeProtoOpTest.testPackedpromote INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedpromote): 0.01s I0310 20:22:57.544095 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedpromote): 0.01s [ FAILED ] DecodeProtoOpTest.testPackedpromote [ RUN ] DecodeProtoOpTest.testPackedragged INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedragged): 0.01s I0310 20:22:57.552472 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedragged): 0.01s [ FAILED ] DecodeProtoOpTest.testPackedragged [ RUN ] DecodeProtoOpTest.testPackedshaped_batch INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedshaped_batch): 0.01s I0310 20:22:57.560965 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedshaped_batch): 0.01s [ FAILED ] DecodeProtoOpTest.testPackedshaped_batch [ RUN ] DecodeProtoOpTest.testPackedsimple INFO:tensorflow:time(__main__.DecodeProtoOpTest.testPackedsimple): 0.01s I0310 20:22:57.568875 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testPackedsimple): 0.01s [ FAILED ] DecodeProtoOpTest.testPackedsimple [ RUN ] DecodeProtoOpTest.testSanitizerGooddefaults INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGooddefaults): 0.01s I0310 20:22:57.578148 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGooddefaults): 0.01s [ FAILED ] DecodeProtoOpTest.testSanitizerGooddefaults [ RUN ] DecodeProtoOpTest.testSanitizerGoodextension INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodextension): 0.01s I0310 20:22:57.586063 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodextension): 0.01s [ OK ] DecodeProtoOpTest.testSanitizerGoodextension [ RUN ] DecodeProtoOpTest.testSanitizerGoodminmax INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodminmax): 0.01s I0310 20:22:57.594895 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodminmax): 0.01s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodminmax [ RUN ] DecodeProtoOpTest.testSanitizerGoodnested INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodnested): 0.01s I0310 20:22:57.602627 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodnested): 0.01s [ OK ] DecodeProtoOpTest.testSanitizerGoodnested [ RUN ] DecodeProtoOpTest.testSanitizerGoodoptional INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodoptional): 0.01s I0310 20:22:57.609555 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodoptional): 0.01s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodoptional [ RUN ] DecodeProtoOpTest.testSanitizerGoodpromote INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodpromote): 0.01s I0310 20:22:57.616782 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodpromote): 0.01s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodpromote [ RUN ] DecodeProtoOpTest.testSanitizerGoodragged INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodragged): 0.01s I0310 20:22:57.623554 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodragged): 0.01s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodragged [ RUN ] DecodeProtoOpTest.testSanitizerGoodshaped_batch INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodshaped_batch): 0.01s I0310 20:22:57.630859 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodshaped_batch): 0.01s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodshaped_batch [ RUN ] DecodeProtoOpTest.testSanitizerGoodsimple INFO:tensorflow:time(__main__.DecodeProtoOpTest.testSanitizerGoodsimple): 0.01s I0310 20:22:57.638728 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testSanitizerGoodsimple): 0.01s [ FAILED ] DecodeProtoOpTest.testSanitizerGoodsimple [ RUN ] DecodeProtoOpTest.testTextdefaults INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextdefaults): 0.01s I0310 20:22:57.650346 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextdefaults): 0.01s [ FAILED ] DecodeProtoOpTest.testTextdefaults [ RUN ] DecodeProtoOpTest.testTextextension INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextextension): 0.01s I0310 20:22:57.661187 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextextension): 0.01s [ OK ] DecodeProtoOpTest.testTextextension [ RUN ] DecodeProtoOpTest.testTextminmax INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextminmax): 0.01s I0310 20:22:57.670680 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextminmax): 0.01s [ FAILED ] DecodeProtoOpTest.testTextminmax [ RUN ] DecodeProtoOpTest.testTextnested INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextnested): 0.01s I0310 20:22:57.677750 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextnested): 0.01s [ OK ] DecodeProtoOpTest.testTextnested [ RUN ] DecodeProtoOpTest.testTextoptional INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextoptional): 0.0s I0310 20:22:57.683827 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextoptional): 0.0s [ FAILED ] DecodeProtoOpTest.testTextoptional [ RUN ] DecodeProtoOpTest.testTextpromote INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextpromote): 0.01s I0310 20:22:57.690783 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextpromote): 0.01s [ FAILED ] DecodeProtoOpTest.testTextpromote [ RUN ] DecodeProtoOpTest.testTextragged INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextragged): 0.0s I0310 20:22:57.696827 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextragged): 0.0s [ FAILED ] DecodeProtoOpTest.testTextragged [ RUN ] DecodeProtoOpTest.testTextshaped_batch INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextshaped_batch): 0.26s I0310 20:22:57.957819 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextshaped_batch): 0.26s [ FAILED ] DecodeProtoOpTest.testTextshaped_batch [ RUN ] DecodeProtoOpTest.testTextsimple INFO:tensorflow:time(__main__.DecodeProtoOpTest.testTextsimple): 0.02s I0310 20:22:57.977740 281473807971200 test_util.py:2467] time(__main__.DecodeProtoOpTest.testTextsimple): 0.02s [ FAILED ] DecodeProtoOpTest.testTextsimple [ RUN ] DecodeProtoOpTest.test_session [ SKIPPED ] DecodeProtoOpTest.test_session ====================================================================== ERROR: testBinaryDisordereddefaults (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisordereddefaults testBinaryDisordereddefaults(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedminmax (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedminmax testBinaryDisorderedminmax(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedoptional (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedoptional testBinaryDisorderedoptional(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedpromote (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedpromote testBinaryDisorderedpromote(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedragged (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedragged testBinaryDisorderedragged(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedshaped_batch (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedshaped_batch testBinaryDisorderedshaped_batch(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryDisorderedsimple (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryDisorderedsimple testBinaryDisorderedsimple(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 223, in testBinaryDisordered self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinarydefaults (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinarydefaults testBinarydefaults(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryminmax (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryminmax testBinaryminmax(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryoptional (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryoptional testBinaryoptional(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinarypromote (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinarypromote testBinarypromote(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryragged (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryragged testBinaryragged(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinaryshaped_batch (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinaryshaped_batch testBinaryshaped_batch(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testBinarysimple (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testBinarysimple testBinarysimple(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 211, in testBinary self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackeddefaults (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackeddefaults testPackeddefaults(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedminmax (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedminmax testPackedminmax(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedoptional (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedoptional testPackedoptional(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedpromote (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedpromote testPackedpromote(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedragged (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedragged testPackedragged(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedshaped_batch (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedshaped_batch testPackedshaped_batch(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testPackedsimple (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testPackedsimple testPackedsimple(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 251, in testPacked self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGooddefaults (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGooddefaults testSanitizerGooddefaults(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodminmax (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodminmax testSanitizerGoodminmax(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodoptional (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodoptional testSanitizerGoodoptional(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodpromote (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodpromote testSanitizerGoodpromote(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodragged (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodragged testSanitizerGoodragged(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodshaped_batch (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodshaped_batch testSanitizerGoodshaped_batch(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testSanitizerGoodsimple (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testSanitizerGoodsimple testSanitizerGoodsimple(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 281, in testSanitizerGood self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextdefaults (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextdefaults testTextdefaults(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextminmax (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextminmax testTextminmax(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextoptional (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextoptional testTextoptional(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextpromote (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextpromote testTextpromote(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextragged (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextragged testTextragged(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextshaped_batch (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextshaped_batch testTextshaped_batch(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ====================================================================== ERROR: testTextsimple (__main__.DecodeProtoOpTest) DecodeProtoOpTest.testTextsimple testTextsimple(values { ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 269, in testText self._runDecodeProtoTests( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 206, in _runDecodeProtoTests self._compareProtos(batch_shape, sizes, fields, field_dict) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 142, in _compareProtos self._compareValues(fd, values.flat, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/proto/decode_proto_op_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/proto/decode_proto_op_test_base.py", line 57, in _compareValues if fd.cpp_type == fd.CPPTYPE_FLOAT: AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'CPPTYPE_FLOAT' ---------------------------------------------------------------------- Ran 48 tests in 0.683s FAILED (errors=35, skipped=1) ================================================================================ ==================== Test output for //tensorflow/python/kernel_tests/data_structures:tensor_array_ops_test_cpu (shard 2 of 10): Running tests under Python 3.9.16: /usr/local/bin/python3 INFO:tensorflow:Using local port 23170 I0310 20:26:36.020175 281473133802368 test_util.py:3797] Using local port 23170 INFO:tensorflow:Using local port 21336 I0310 20:26:36.020888 281473133802368 test_util.py:3797] Using local port 21336 INFO:tensorflow:Using local port 20627 I0310 20:26:36.021311 281473133802368 test_util.py:3797] Using local port 20627 2023-03-10 20:26:36.149593: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:23170 2023-03-10 20:26:36.323045: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:450] Started server with target: grpc://localhost:21336 E0310 20:26:36.324312946 1898842 server_chttp2.cc:40] {"created":"@1678479996.324177837","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1678479996.324172362","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1678479996.324139258","description":"Unable to configure socket","fd":11,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1678479996.324106235","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1678479996.324171197","description":"Unable to configure socket","fd":11,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1678479996.324162681","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} 2023-03-10 20:26:36.324423: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:601] UNKNOWN: Could not start gRPC server [ FAILED ] setUpClass (__main__.TensorArrayTest) ====================================================================== ERROR: setUpClass (__main__.TensorArrayTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/data_structures/tensor_array_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/data_structures/tensor_array_ops_test.py", line 75, in setUpClass cls._workers, _ = test.create_local_cluster(num_workers=3, num_ps=0) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/data_structures/tensor_array_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3867, in create_local_cluster workers = [ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/data_structures/tensor_array_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3868, in server_lib.Server( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/data_structures/tensor_array_ops_test_cpu.runfiles/org_tensorflow/tensorflow/python/training/server_lib.py", line 144, in __init__ self._server = c_api.TF_NewServer(self._server_def.SerializeToString()) tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server ---------------------------------------------------------------------- Ran 0 tests in 0.395s FAILED (errors=1) ================================================================================ ==================== Test output for //tensorflow/python/util/protobuf:protobuf_compare_test: Running tests under Python 3.9.16: /usr/local/bin/python3 [ RUN ] AssertTest.testAssertEqualWithStringArg [ OK ] AssertTest.testAssertEqualWithStringArg [ RUN ] AssertTest.testCheckInitialized [ FAILED ] AssertTest.testCheckInitialized [ RUN ] AssertTest.testLargeProtoData [ OK ] AssertTest.testLargeProtoData [ RUN ] AssertTest.testMessage [ OK ] AssertTest.testMessage [ RUN ] AssertTest.testMsgPassdown [ OK ] AssertTest.testMsgPassdown [ RUN ] AssertTest.testNormalizesFloat [ OK ] AssertTest.testNormalizesFloat [ RUN ] AssertTest.testNormalizesNumbers [ OK ] AssertTest.testNormalizesNumbers [ RUN ] AssertTest.testPrimitives [ OK ] AssertTest.testPrimitives [ RUN ] AssertTest.testRepeatedMessage [ OK ] AssertTest.testRepeatedMessage [ RUN ] AssertTest.testRepeatedPrimitives [ OK ] AssertTest.testRepeatedPrimitives [ RUN ] MixinTests.testAssertEqualWithStringArg [ OK ] MixinTests.testAssertEqualWithStringArg [ RUN ] NormalizeNumbersTest.testNormalizesDoubles [ OK ] NormalizeNumbersTest.testNormalizesDoubles [ RUN ] NormalizeNumbersTest.testNormalizesFloats [ OK ] NormalizeNumbersTest.testNormalizesFloats [ RUN ] NormalizeNumbersTest.testNormalizesInts [ OK ] NormalizeNumbersTest.testNormalizesInts [ RUN ] NormalizeNumbersTest.testNormalizesMaps [ OK ] NormalizeNumbersTest.testNormalizesMaps [ RUN ] NormalizeNumbersTest.testNormalizesRepeatedFloats [ OK ] NormalizeNumbersTest.testNormalizesRepeatedFloats [ RUN ] NormalizeNumbersTest.testNormalizesRepeatedInts [ OK ] NormalizeNumbersTest.testNormalizesRepeatedInts [ RUN ] ProtoEqTest.testEmpty [ OK ] ProtoEqTest.testEmpty [ RUN ] ProtoEqTest.testIsClose [ OK ] ProtoEqTest.testIsClose [ RUN ] ProtoEqTest.testMessage [ OK ] ProtoEqTest.testMessage [ RUN ] ProtoEqTest.testNans [ OK ] ProtoEqTest.testNans [ RUN ] ProtoEqTest.testNansMap [ OK ] ProtoEqTest.testNansMap [ RUN ] ProtoEqTest.testNansRepeatedField [ OK ] ProtoEqTest.testNansRepeatedField [ RUN ] ProtoEqTest.testNestedMessage [ OK ] ProtoEqTest.testNestedMessage [ RUN ] ProtoEqTest.testPrimitiveFields [ OK ] ProtoEqTest.testPrimitiveFields [ RUN ] ProtoEqTest.testPrimitives [ OK ] ProtoEqTest.testPrimitives [ RUN ] ProtoEqTest.testRepeatedPrimitives [ OK ] ProtoEqTest.testRepeatedPrimitives [ RUN ] ProtoEqTest.testTagOrder [ OK ] ProtoEqTest.testTagOrder ====================================================================== ERROR: testCheckInitialized (__main__.AssertTest) AssertTest.testCheckInitialized ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/util/protobuf/protobuf_compare_test.runfiles/org_tensorflow/tensorflow/python/util/protobuf/compare_test.py", line 324, in testCheckInitialized b = copy.deepcopy(a) File "/usr/lib/python3.9/copy.py", line 153, in deepcopy y = copier(memo) File "/usr/local/lib/python3.9/dist-packages/google/protobuf/message.py", line 83, in __deepcopy__ clone.MergeFrom(self) google.protobuf.message.EncodeError: Message compare_test.Labeled is missing required fields: required ---------------------------------------------------------------------- Ran 28 tests in 1.691s FAILED (errors=1) ================================================================================ ==================== Test output for //tensorflow/python/util/protobuf:protobuf_compare_test: Running tests under Python 3.9.16: /usr/local/bin/python3 [ RUN ] AssertTest.testAssertEqualWithStringArg [ OK ] AssertTest.testAssertEqualWithStringArg [ RUN ] AssertTest.testCheckInitialized [ FAILED ] AssertTest.testCheckInitialized [ RUN ] AssertTest.testLargeProtoData [ OK ] AssertTest.testLargeProtoData [ RUN ] AssertTest.testMessage [ OK ] AssertTest.testMessage [ RUN ] AssertTest.testMsgPassdown [ OK ] AssertTest.testMsgPassdown [ RUN ] AssertTest.testNormalizesFloat [ OK ] AssertTest.testNormalizesFloat [ RUN ] AssertTest.testNormalizesNumbers [ OK ] AssertTest.testNormalizesNumbers [ RUN ] AssertTest.testPrimitives [ OK ] AssertTest.testPrimitives [ RUN ] AssertTest.testRepeatedMessage [ OK ] AssertTest.testRepeatedMessage [ RUN ] AssertTest.testRepeatedPrimitives [ OK ] AssertTest.testRepeatedPrimitives [ RUN ] MixinTests.testAssertEqualWithStringArg [ OK ] MixinTests.testAssertEqualWithStringArg [ RUN ] NormalizeNumbersTest.testNormalizesDoubles [ OK ] NormalizeNumbersTest.testNormalizesDoubles [ RUN ] NormalizeNumbersTest.testNormalizesFloats [ OK ] NormalizeNumbersTest.testNormalizesFloats [ RUN ] NormalizeNumbersTest.testNormalizesInts [ OK ] NormalizeNumbersTest.testNormalizesInts [ RUN ] NormalizeNumbersTest.testNormalizesMaps [ OK ] NormalizeNumbersTest.testNormalizesMaps [ RUN ] NormalizeNumbersTest.testNormalizesRepeatedFloats [ OK ] NormalizeNumbersTest.testNormalizesRepeatedFloats [ RUN ] NormalizeNumbersTest.testNormalizesRepeatedInts [ OK ] NormalizeNumbersTest.testNormalizesRepeatedInts [ RUN ] ProtoEqTest.testEmpty [ OK ] ProtoEqTest.testEmpty [ RUN ] ProtoEqTest.testIsClose [ OK ] ProtoEqTest.testIsClose [ RUN ] ProtoEqTest.testMessage [ OK ] ProtoEqTest.testMessage [ RUN ] ProtoEqTest.testNans [ OK ] ProtoEqTest.testNans [ RUN ] ProtoEqTest.testNansMap [ OK ] ProtoEqTest.testNansMap [ RUN ] ProtoEqTest.testNansRepeatedField [ OK ] ProtoEqTest.testNansRepeatedField [ RUN ] ProtoEqTest.testNestedMessage [ OK ] ProtoEqTest.testNestedMessage [ RUN ] ProtoEqTest.testPrimitiveFields [ OK ] ProtoEqTest.testPrimitiveFields [ RUN ] ProtoEqTest.testPrimitives [ OK ] ProtoEqTest.testPrimitives [ RUN ] ProtoEqTest.testRepeatedPrimitives [ OK ] ProtoEqTest.testRepeatedPrimitives [ RUN ] ProtoEqTest.testTagOrder [ OK ] ProtoEqTest.testTagOrder ====================================================================== ERROR: testCheckInitialized (__main__.AssertTest) AssertTest.testCheckInitialized ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/util/protobuf/protobuf_compare_test.runfiles/org_tensorflow/tensorflow/python/util/protobuf/compare_test.py", line 324, in testCheckInitialized b = copy.deepcopy(a) File "/usr/lib/python3.9/copy.py", line 153, in deepcopy y = copier(memo) File "/usr/local/lib/python3.9/dist-packages/google/protobuf/message.py", line 83, in __deepcopy__ clone.MergeFrom(self) google.protobuf.message.EncodeError: Message compare_test.Labeled is missing required fields: required ---------------------------------------------------------------------- Ran 28 tests in 1.022s FAILED (errors=1) ================================================================================ ==================== Test output for //tensorflow/python/util/protobuf:protobuf_compare_test: Running tests under Python 3.9.16: /usr/local/bin/python3 [ RUN ] AssertTest.testAssertEqualWithStringArg [ OK ] AssertTest.testAssertEqualWithStringArg [ RUN ] AssertTest.testCheckInitialized [ FAILED ] AssertTest.testCheckInitialized [ RUN ] AssertTest.testLargeProtoData [ OK ] AssertTest.testLargeProtoData [ RUN ] AssertTest.testMessage [ OK ] AssertTest.testMessage [ RUN ] AssertTest.testMsgPassdown [ OK ] AssertTest.testMsgPassdown [ RUN ] AssertTest.testNormalizesFloat [ OK ] AssertTest.testNormalizesFloat [ RUN ] AssertTest.testNormalizesNumbers [ OK ] AssertTest.testNormalizesNumbers [ RUN ] AssertTest.testPrimitives [ OK ] AssertTest.testPrimitives [ RUN ] AssertTest.testRepeatedMessage [ OK ] AssertTest.testRepeatedMessage [ RUN ] AssertTest.testRepeatedPrimitives [ OK ] AssertTest.testRepeatedPrimitives [ RUN ] MixinTests.testAssertEqualWithStringArg [ OK ] MixinTests.testAssertEqualWithStringArg [ RUN ] NormalizeNumbersTest.testNormalizesDoubles [ OK ] NormalizeNumbersTest.testNormalizesDoubles [ RUN ] NormalizeNumbersTest.testNormalizesFloats [ OK ] NormalizeNumbersTest.testNormalizesFloats [ RUN ] NormalizeNumbersTest.testNormalizesInts [ OK ] NormalizeNumbersTest.testNormalizesInts [ RUN ] NormalizeNumbersTest.testNormalizesMaps [ OK ] NormalizeNumbersTest.testNormalizesMaps [ RUN ] NormalizeNumbersTest.testNormalizesRepeatedFloats [ OK ] NormalizeNumbersTest.testNormalizesRepeatedFloats [ RUN ] NormalizeNumbersTest.testNormalizesRepeatedInts [ OK ] NormalizeNumbersTest.testNormalizesRepeatedInts [ RUN ] ProtoEqTest.testEmpty [ OK ] ProtoEqTest.testEmpty [ RUN ] ProtoEqTest.testIsClose [ OK ] ProtoEqTest.testIsClose [ RUN ] ProtoEqTest.testMessage [ OK ] ProtoEqTest.testMessage [ RUN ] ProtoEqTest.testNans [ OK ] ProtoEqTest.testNans [ RUN ] ProtoEqTest.testNansMap [ OK ] ProtoEqTest.testNansMap [ RUN ] ProtoEqTest.testNansRepeatedField [ OK ] ProtoEqTest.testNansRepeatedField [ RUN ] ProtoEqTest.testNestedMessage [ OK ] ProtoEqTest.testNestedMessage [ RUN ] ProtoEqTest.testPrimitiveFields [ OK ] ProtoEqTest.testPrimitiveFields [ RUN ] ProtoEqTest.testPrimitives [ OK ] ProtoEqTest.testPrimitives [ RUN ] ProtoEqTest.testRepeatedPrimitives [ OK ] ProtoEqTest.testRepeatedPrimitives [ RUN ] ProtoEqTest.testTagOrder [ OK ] ProtoEqTest.testTagOrder ====================================================================== ERROR: testCheckInitialized (__main__.AssertTest) AssertTest.testCheckInitialized ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/util/protobuf/protobuf_compare_test.runfiles/org_tensorflow/tensorflow/python/util/protobuf/compare_test.py", line 324, in testCheckInitialized b = copy.deepcopy(a) File "/usr/lib/python3.9/copy.py", line 153, in deepcopy y = copier(memo) File "/usr/local/lib/python3.9/dist-packages/google/protobuf/message.py", line 83, in __deepcopy__ clone.MergeFrom(self) google.protobuf.message.EncodeError: Message compare_test.Labeled is missing required fields: required ---------------------------------------------------------------------- Ran 28 tests in 1.033s FAILED (errors=1) ================================================================================ ==================== Test output for //tensorflow/tools/api/tests:api_compatibility_test: Running tests under Python 3.9.16: /usr/local/bin/python3 [ RUN ] ApiCompatibilityTest.testAPIBackwardsCompatibility ERROR:tensorflow:TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. E0310 20:36:02.940023 281472836531072 api_compatibility_test.py:370] TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. ERROR:tensorflow:1 differences found between API and golden. E0310 20:36:02.940366 281472836531072 api_compatibility_test.py:371] 1 differences found between API and golden. ERROR:tensorflow: Change detected in python object: tensorflow.train. E0310 20:36:02.940526 281472836531072 api_compatibility_test.py:392] Change detected in python object: tensorflow.train. ERROR:tensorflow: path: "tensorflow.train" tf_module { member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointView" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "TrackableView" mtype: "" } member { name: "experimental" mtype: "" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } } E0310 20:36:02.940671 281472836531072 api_compatibility_test.py:393] path: "tensorflow.train" tf_module { member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointView" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "TrackableView" mtype: "" } member { name: "experimental" mtype: "" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } } INFO:tensorflow:time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibility): 2.43s I0310 20:36:02.940945 281472836531072 test_util.py:2467] time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibility): 2.43s [ FAILED ] ApiCompatibilityTest.testAPIBackwardsCompatibility [ RUN ] ApiCompatibilityTest.testAPIBackwardsCompatibilityV1 WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/python/util/decorator_utils.py:153: GraphKeys.VARIABLES (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.GraphKeys.GLOBAL_VARIABLES` instead. W0310 20:36:03.036035 281472836531072 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/python/util/decorator_utils.py:153: GraphKeys.VARIABLES (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.GraphKeys.GLOBAL_VARIABLES` instead. ERROR:tensorflow:TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. E0310 20:36:05.804020 281472836531072 api_compatibility_test.py:370] TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. ERROR:tensorflow:4 differences found between API and golden. E0310 20:36:05.804332 281472836531072 api_compatibility_test.py:371] 4 differences found between API and golden. ERROR:tensorflow: Change detected in python object: tensorflow.profiler. E0310 20:36:05.804494 281472836531072 api_compatibility_test.py:392] Change detected in python object: tensorflow.profiler. ERROR:tensorflow: path: "tensorflow.profiler" tf_module { member { name: "AdviceProto" - mtype: "" + mtype: "" } member { name: "GraphNodeProto" - mtype: "" + mtype: "" } member { name: "MultiGraphNodeProto" - mtype: "" + mtype: "" } member { name: "OpLogProto" - mtype: "" + mtype: "" } member { name: "ProfileOptionBuilder" mtype: "" } member { name: "Profiler" mtype: "" } member_method { name: "advise" argspec: "args=[\'graph\', \'run_meta\', \'options\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'0\'], " } member_method { name: "profile" argspec: "args=[\'graph\', \'run_meta\', \'op_log\', \'cmd\', \'options\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'scope\', \'0\'], " } member_method { name: "write_op_log" argspec: "args=[\'graph\', \'log_dir\', \'op_log\', \'run_meta\', \'add_trace\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\'], " } } E0310 20:36:05.804638 281472836531072 api_compatibility_test.py:393] path: "tensorflow.profiler" tf_module { member { name: "AdviceProto" - mtype: "" + mtype: "" } member { name: "GraphNodeProto" - mtype: "" + mtype: "" } member { name: "MultiGraphNodeProto" - mtype: "" + mtype: "" } member { name: "OpLogProto" - mtype: "" + mtype: "" } member { name: "ProfileOptionBuilder" mtype: "" } member { name: "Profiler" mtype: "" } member_method { name: "advise" argspec: "args=[\'graph\', \'run_meta\', \'options\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'0\'], " } member_method { name: "profile" argspec: "args=[\'graph\', \'run_meta\', \'op_log\', \'cmd\', \'options\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'scope\', \'0\'], " } member_method { name: "write_op_log" argspec: "args=[\'graph\', \'log_dir\', \'op_log\', \'run_meta\', \'add_trace\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\'], " } } ERROR:tensorflow: Change detected in python object: tensorflow.train. E0310 20:36:05.804792 281472836531072 api_compatibility_test.py:392] Change detected in python object: tensorflow.train. ERROR:tensorflow: path: "tensorflow.train" tf_module { member { name: "AdadeltaOptimizer" mtype: "" } member { name: "AdagradDAOptimizer" mtype: "" } member { name: "AdagradOptimizer" mtype: "" } member { name: "AdamOptimizer" mtype: "" } member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointSaverHook" mtype: "" } member { name: "CheckpointSaverListener" mtype: "" } member { name: "ChiefSessionCreator" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FeedFnHook" mtype: "" } member { name: "FinalOpsHook" mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "FtrlOptimizer" mtype: "" } member { name: "GlobalStepWaiterHook" mtype: "" } member { name: "GradientDescentOptimizer" mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "LoggingTensorHook" mtype: "" } member { name: "LooperThread" mtype: "" } member { name: "MomentumOptimizer" mtype: "" } member { name: "MonitoredSession" mtype: "" } member { name: "NanLossDuringTrainingError" mtype: "" } member { name: "NanTensorHook" mtype: "" } member { name: "Optimizer" mtype: "" } member { name: "ProfilerHook" mtype: "" } member { name: "ProximalAdagradOptimizer" mtype: "" } member { name: "ProximalGradientDescentOptimizer" mtype: "" } member { name: "QueueRunner" mtype: "" } member { name: "RMSPropOptimizer" mtype: "" } member { name: "Saver" mtype: "" } member { name: "SaverDef" - mtype: "" + mtype: "" } member { name: "Scaffold" mtype: "" } member { name: "SecondOrStepTimer" mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "Server" mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "SessionCreator" mtype: "" } member { name: "SessionManager" mtype: "" } member { name: "SessionRunArgs" mtype: "" } member { name: "SessionRunContext" mtype: "" } member { name: "SessionRunHook" mtype: "" } member { name: "SessionRunValues" mtype: "" } member { name: "SingularMonitoredSession" mtype: "" } member { name: "StepCounterHook" mtype: "" } member { name: "StopAtStepHook" mtype: "" } member { name: "SummarySaverHook" mtype: "" } member { name: "Supervisor" mtype: "" } member { name: "SyncReplicasOptimizer" mtype: "" } member { name: "VocabInfo" mtype: "" } member { name: "WorkerSessionCreator" mtype: "" } member { name: "experimental" mtype: "" } member { name: "queue_runner" mtype: "" } member_method { name: "MonitoredTrainingSession" argspec: "args=[\'master\', \'is_chief\', \'checkpoint_dir\', \'scaffold\', \'hooks\', \'chief_only_hooks\', \'save_checkpoint_secs\', \'save_summaries_steps\', \'save_summaries_secs\', \'config\', \'stop_grace_period_secs\', \'log_step_count_steps\', \'max_wait_secs\', \'save_checkpoint_steps\', \'summary_dir\', \'save_graph_def\'], varargs=None, keywords=None, defaults=[\'\', \'True\', \'None\', \'None\', \'None\', \'None\', \'\', \'\', \'\', \'None\', \'120\', \'100\', \'7200\', \'\', \'None\', \'True\'], " } member_method { name: "NewCheckpointReader" argspec: "args=[\'filepattern\'], varargs=None, keywords=None, defaults=None" } member_method { name: "add_queue_runner" argspec: "args=[\'qr\', \'collection\'], varargs=None, keywords=None, defaults=[\'queue_runners\'], " } member_method { name: "assert_global_step" argspec: "args=[\'global_step_tensor\'], varargs=None, keywords=None, defaults=None" } member_method { name: "basic_train_loop" argspec: "args=[\'supervisor\', \'train_step_fn\', \'args\', \'kwargs\', \'master\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'\'], " } member_method { name: "batch" argspec: "args=[\'tensors\', \'batch_size\', \'num_threads\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "checkpoint_exists" argspec: "args=[\'checkpoint_prefix\'], varargs=None, keywords=None, defaults=None" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'alpha\', \'name\'], varargs=None, keywords=None, defaults=[\'0.0\', \'None\'], " } member_method { name: "cosine_decay_restarts" argspec: "args=[\'learning_rate\', \'global_step\', \'first_decay_steps\', \'t_mul\', \'m_mul\', \'alpha\', \'name\'], varargs=None, keywords=None, defaults=[\'2.0\', \'1.0\', \'0.0\', \'None\'], " } member_method { name: "create_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "do_quantize_training_on_graphdef" argspec: "args=[\'input_graph\', \'num_bits\'], varargs=None, keywords=None, defaults=None" } member_method { name: "exponential_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "export_meta_graph" argspec: "args=[\'filename\', \'meta_info_def\', \'graph_def\', \'saver_def\', \'collection_list\', \'as_text\', \'graph\', \'export_scope\', \'clear_devices\', \'clear_extraneous_savers\', \'strip_default_attrs\', \'save_debug_info\'], varargs=None, keywords=kwargs, defaults=[\'None\', \'None\', \'None\', \'None\', \'None\', \'False\', \'None\', \'None\', \'False\', \'False\', \'False\', \'False\'], " } member_method { name: "generate_checkpoint_state_proto" argspec: "args=[\'save_dir\', \'model_checkpoint_path\', \'all_model_checkpoint_paths\', \'all_model_checkpoint_timestamps\', \'last_preserved_timestamp\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_mtimes" argspec: "args=[\'checkpoint_prefixes\'], varargs=None, keywords=None, defaults=None" } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "get_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "get_or_create_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "global_step" argspec: "args=[\'sess\', \'global_step_tensor\'], varargs=None, keywords=None, defaults=None" } member_method { name: "import_meta_graph" argspec: "args=[\'meta_graph_or_file\', \'clear_devices\', \'import_scope\'], varargs=None, keywords=kwargs, defaults=[\'False\', \'None\'], " } member_method { name: "init_from_checkpoint" argspec: "args=[\'ckpt_dir_or_file\', \'assignment_map\'], varargs=None, keywords=None, defaults=None" } member_method { name: "input_producer" argspec: "args=[\'input_tensor\', \'element_shape\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'summary_name\', \'name\', \'cancel_op\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\', \'None\', \'32\', \'None\', \'None\', \'None\', \'None\'], " } member_method { name: "inverse_time_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "limit_epochs" argspec: "args=[\'tensor\', \'num_epochs\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "linear_cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'num_periods\', \'alpha\', \'beta\', \'name\'], varargs=None, keywords=None, defaults=[\'0.5\', \'0.0\', \'0.001\', \'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } member_method { name: "match_filenames_once" argspec: "args=[\'pattern\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "maybe_batch" argspec: "args=[\'tensors\', \'keep_input\', \'batch_size\', \'num_threads\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_batch_join" argspec: "args=[\'tensors_list\', \'keep_input\', \'batch_size\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_shuffle_batch" argspec: "args=[\'tensors\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'keep_input\', \'num_threads\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_shuffle_batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'keep_input\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "natural_exp_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "noisy_linear_cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'initial_variance\', \'variance_decay\', \'num_periods\', \'alpha\', \'beta\', \'name\'], varargs=None, keywords=None, defaults=[\'1.0\', \'0.55\', \'0.5\', \'0.0\', \'0.001\', \'None\'], " } member_method { name: "piecewise_constant" argspec: "args=[\'x\', \'boundaries\', \'values\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "piecewise_constant_decay" argspec: "args=[\'x\', \'boundaries\', \'values\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "polynomial_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'end_learning_rate\', \'power\', \'cycle\', \'name\'], varargs=None, keywords=None, defaults=[\'0.0001\', \'1.0\', \'False\', \'None\'], " } member_method { name: "range_input_producer" argspec: "args=[\'limit\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\'], " } member_method { name: "remove_checkpoint" argspec: "args=[\'checkpoint_prefix\', \'checkpoint_format_version\', \'meta_graph_suffix\'], varargs=None, keywords=None, defaults=[\'2\', \'meta\'], " } member_method { name: "replica_device_setter" argspec: "args=[\'ps_tasks\', \'ps_device\', \'worker_device\', \'merge_devices\', \'cluster\', \'ps_ops\', \'ps_strategy\'], varargs=None, keywords=None, defaults=[\'0\', \'/job:ps\', \'/job:worker\', \'True\', \'None\', \'None\', \'None\'], " } member_method { name: "sdca_fprint" argspec: "args=[\'input\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "sdca_optimizer" argspec: "args=[\'sparse_example_indices\', \'sparse_feature_indices\', \'sparse_feature_values\', \'dense_features\', \'example_weights\', \'example_labels\', \'sparse_indices\', \'sparse_weights\', \'dense_weights\', \'example_state_data\', \'loss_type\', \'l1\', \'l2\', \'num_loss_partitions\', \'num_inner_iterations\', \'adaptative\', \'name\'], varargs=None, keywords=None, defaults=[\'True\', \'None\'], " } member_method { name: "sdca_shrink_l1" argspec: "args=[\'weights\', \'l1\', \'l2\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "shuffle_batch" argspec: "args=[\'tensors\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'num_threads\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "shuffle_batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "slice_input_producer" argspec: "args=[\'tensor_list\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\'], " } member_method { name: "start_queue_runners" argspec: "args=[\'sess\', \'coord\', \'daemon\', \'start\', \'collection\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\', \'True\', \'queue_runners\'], " } member_method { name: "string_input_producer" argspec: "args=[\'string_tensor\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\', \'cancel_op\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\', \'None\'], " } member_method { name: "summary_iterator" argspec: "args=[\'path\'], varargs=None, keywords=None, defaults=None" } member_method { name: "update_checkpoint_state" argspec: "args=[\'save_dir\', \'model_checkpoint_path\', \'all_model_checkpoint_paths\', \'latest_filename\', \'all_model_checkpoint_timestamps\', \'last_preserved_timestamp\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'None\'], " } member_method { name: "warm_start" argspec: "args=[\'ckpt_to_initialize_from\', \'vars_to_warm_start\', \'var_name_to_vocab_info\', \'var_name_to_prev_var_name\'], varargs=None, keywords=None, defaults=[\'.*\', \'None\', \'None\'], " } member_method { name: "write_graph" argspec: "args=[\'graph_or_graph_def\', \'logdir\', \'name\', \'as_text\'], varargs=None, keywords=None, defaults=[\'True\'], " } } E0310 20:36:05.804936 281472836531072 api_compatibility_test.py:393] path: "tensorflow.train" tf_module { member { name: "AdadeltaOptimizer" mtype: "" } member { name: "AdagradDAOptimizer" mtype: "" } member { name: "AdagradOptimizer" mtype: "" } member { name: "AdamOptimizer" mtype: "" } member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointSaverHook" mtype: "" } member { name: "CheckpointSaverListener" mtype: "" } member { name: "ChiefSessionCreator" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FeedFnHook" mtype: "" } member { name: "FinalOpsHook" mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "FtrlOptimizer" mtype: "" } member { name: "GlobalStepWaiterHook" mtype: "" } member { name: "GradientDescentOptimizer" mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "LoggingTensorHook" mtype: "" } member { name: "LooperThread" mtype: "" } member { name: "MomentumOptimizer" mtype: "" } member { name: "MonitoredSession" mtype: "" } member { name: "NanLossDuringTrainingError" mtype: "" } member { name: "NanTensorHook" mtype: "" } member { name: "Optimizer" mtype: "" } member { name: "ProfilerHook" mtype: "" } member { name: "ProximalAdagradOptimizer" mtype: "" } member { name: "ProximalGradientDescentOptimizer" mtype: "" } member { name: "QueueRunner" mtype: "" } member { name: "RMSPropOptimizer" mtype: "" } member { name: "Saver" mtype: "" } member { name: "SaverDef" - mtype: "" + mtype: "" } member { name: "Scaffold" mtype: "" } member { name: "SecondOrStepTimer" mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "Server" mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "SessionCreator" mtype: "" } member { name: "SessionManager" mtype: "" } member { name: "SessionRunArgs" mtype: "" } member { name: "SessionRunContext" mtype: "" } member { name: "SessionRunHook" mtype: "" } member { name: "SessionRunValues" mtype: "" } member { name: "SingularMonitoredSession" mtype: "" } member { name: "StepCounterHook" mtype: "" } member { name: "StopAtStepHook" mtype: "" } member { name: "SummarySaverHook" mtype: "" } member { name: "Supervisor" mtype: "" } member { name: "SyncReplicasOptimizer" mtype: "" } member { name: "VocabInfo" mtype: "" } member { name: "WorkerSessionCreator" mtype: "" } member { name: "experimental" mtype: "" } member { name: "queue_runner" mtype: "" } member_method { name: "MonitoredTrainingSession" argspec: "args=[\'master\', \'is_chief\', \'checkpoint_dir\', \'scaffold\', \'hooks\', \'chief_only_hooks\', \'save_checkpoint_secs\', \'save_summaries_steps\', \'save_summaries_secs\', \'config\', \'stop_grace_period_secs\', \'log_step_count_steps\', \'max_wait_secs\', \'save_checkpoint_steps\', \'summary_dir\', \'save_graph_def\'], varargs=None, keywords=None, defaults=[\'\', \'True\', \'None\', \'None\', \'None\', \'None\', \'\', \'\', \'\', \'None\', \'120\', \'100\', \'7200\', \'\', \'None\', \'True\'], " } member_method { name: "NewCheckpointReader" argspec: "args=[\'filepattern\'], varargs=None, keywords=None, defaults=None" } member_method { name: "add_queue_runner" argspec: "args=[\'qr\', \'collection\'], varargs=None, keywords=None, defaults=[\'queue_runners\'], " } member_method { name: "assert_global_step" argspec: "args=[\'global_step_tensor\'], varargs=None, keywords=None, defaults=None" } member_method { name: "basic_train_loop" argspec: "args=[\'supervisor\', \'train_step_fn\', \'args\', \'kwargs\', \'master\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'\'], " } member_method { name: "batch" argspec: "args=[\'tensors\', \'batch_size\', \'num_threads\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "checkpoint_exists" argspec: "args=[\'checkpoint_prefix\'], varargs=None, keywords=None, defaults=None" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'alpha\', \'name\'], varargs=None, keywords=None, defaults=[\'0.0\', \'None\'], " } member_method { name: "cosine_decay_restarts" argspec: "args=[\'learning_rate\', \'global_step\', \'first_decay_steps\', \'t_mul\', \'m_mul\', \'alpha\', \'name\'], varargs=None, keywords=None, defaults=[\'2.0\', \'1.0\', \'0.0\', \'None\'], " } member_method { name: "create_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "do_quantize_training_on_graphdef" argspec: "args=[\'input_graph\', \'num_bits\'], varargs=None, keywords=None, defaults=None" } member_method { name: "exponential_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "export_meta_graph" argspec: "args=[\'filename\', \'meta_info_def\', \'graph_def\', \'saver_def\', \'collection_list\', \'as_text\', \'graph\', \'export_scope\', \'clear_devices\', \'clear_extraneous_savers\', \'strip_default_attrs\', \'save_debug_info\'], varargs=None, keywords=kwargs, defaults=[\'None\', \'None\', \'None\', \'None\', \'None\', \'False\', \'None\', \'None\', \'False\', \'False\', \'False\', \'False\'], " } member_method { name: "generate_checkpoint_state_proto" argspec: "args=[\'save_dir\', \'model_checkpoint_path\', \'all_model_checkpoint_paths\', \'all_model_checkpoint_timestamps\', \'last_preserved_timestamp\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_mtimes" argspec: "args=[\'checkpoint_prefixes\'], varargs=None, keywords=None, defaults=None" } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "get_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "get_or_create_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "global_step" argspec: "args=[\'sess\', \'global_step_tensor\'], varargs=None, keywords=None, defaults=None" } member_method { name: "import_meta_graph" argspec: "args=[\'meta_graph_or_file\', \'clear_devices\', \'import_scope\'], varargs=None, keywords=kwargs, defaults=[\'False\', \'None\'], " } member_method { name: "init_from_checkpoint" argspec: "args=[\'ckpt_dir_or_file\', \'assignment_map\'], varargs=None, keywords=None, defaults=None" } member_method { name: "input_producer" argspec: "args=[\'input_tensor\', \'element_shape\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'summary_name\', \'name\', \'cancel_op\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\', \'None\', \'32\', \'None\', \'None\', \'None\', \'None\'], " } member_method { name: "inverse_time_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "limit_epochs" argspec: "args=[\'tensor\', \'num_epochs\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "linear_cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'num_periods\', \'alpha\', \'beta\', \'name\'], varargs=None, keywords=None, defaults=[\'0.5\', \'0.0\', \'0.001\', \'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } member_method { name: "match_filenames_once" argspec: "args=[\'pattern\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "maybe_batch" argspec: "args=[\'tensors\', \'keep_input\', \'batch_size\', \'num_threads\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_batch_join" argspec: "args=[\'tensors_list\', \'keep_input\', \'batch_size\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_shuffle_batch" argspec: "args=[\'tensors\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'keep_input\', \'num_threads\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_shuffle_batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'keep_input\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "natural_exp_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "noisy_linear_cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'initial_variance\', \'variance_decay\', \'num_periods\', \'alpha\', \'beta\', \'name\'], varargs=None, keywords=None, defaults=[\'1.0\', \'0.55\', \'0.5\', \'0.0\', \'0.001\', \'None\'], " } member_method { name: "piecewise_constant" argspec: "args=[\'x\', \'boundaries\', \'values\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "piecewise_constant_decay" argspec: "args=[\'x\', \'boundaries\', \'values\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "polynomial_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'end_learning_rate\', \'power\', \'cycle\', \'name\'], varargs=None, keywords=None, defaults=[\'0.0001\', \'1.0\', \'False\', \'None\'], " } member_method { name: "range_input_producer" argspec: "args=[\'limit\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\'], " } member_method { name: "remove_checkpoint" argspec: "args=[\'checkpoint_prefix\', \'checkpoint_format_version\', \'meta_graph_suffix\'], varargs=None, keywords=None, defaults=[\'2\', \'meta\'], " } member_method { name: "replica_device_setter" argspec: "args=[\'ps_tasks\', \'ps_device\', \'worker_device\', \'merge_devices\', \'cluster\', \'ps_ops\', \'ps_strategy\'], varargs=None, keywords=None, defaults=[\'0\', \'/job:ps\', \'/job:worker\', \'True\', \'None\', \'None\', \'None\'], " } member_method { name: "sdca_fprint" argspec: "args=[\'input\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "sdca_optimizer" argspec: "args=[\'sparse_example_indices\', \'sparse_feature_indices\', \'sparse_feature_values\', \'dense_features\', \'example_weights\', \'example_labels\', \'sparse_indices\', \'sparse_weights\', \'dense_weights\', \'example_state_data\', \'loss_type\', \'l1\', \'l2\', \'num_loss_partitions\', \'num_inner_iterations\', \'adaptative\', \'name\'], varargs=None, keywords=None, defaults=[\'True\', \'None\'], " } member_method { name: "sdca_shrink_l1" argspec: "args=[\'weights\', \'l1\', \'l2\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "shuffle_batch" argspec: "args=[\'tensors\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'num_threads\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "shuffle_batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "slice_input_producer" argspec: "args=[\'tensor_list\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\'], " } member_method { name: "start_queue_runners" argspec: "args=[\'sess\', \'coord\', \'daemon\', \'start\', \'collection\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\', \'True\', \'queue_runners\'], " } member_method { name: "string_input_producer" argspec: "args=[\'string_tensor\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\', \'cancel_op\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\', \'None\'], " } member_method { name: "summary_iterator" argspec: "args=[\'path\'], varargs=None, keywords=None, defaults=None" } member_method { name: "update_checkpoint_state" argspec: "args=[\'save_dir\', \'model_checkpoint_path\', \'all_model_checkpoint_paths\', \'latest_filename\', \'all_model_checkpoint_timestamps\', \'last_preserved_timestamp\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'None\'], " } member_method { name: "warm_start" argspec: "args=[\'ckpt_to_initialize_from\', \'vars_to_warm_start\', \'var_name_to_vocab_info\', \'var_name_to_prev_var_name\'], varargs=None, keywords=None, defaults=[\'.*\', \'None\', \'None\'], " } member_method { name: "write_graph" argspec: "args=[\'graph_or_graph_def\', \'logdir\', \'name\', \'as_text\'], varargs=None, keywords=None, defaults=[\'True\'], " } } ERROR:tensorflow: Change detected in python object: tensorflow. E0310 20:36:05.805130 281472836531072 api_compatibility_test.py:392] Change detected in python object: tensorflow. ERROR:tensorflow: None : --- +++ @@ -10,7 +10,7 @@ } member { name: "AttrValue" - mtype: "" + mtype: "" } member { name: "COMPILER_VERSION" @@ -34,7 +34,7 @@ } member { name: "ConfigProto" - mtype: "" + mtype: "" } member { name: "CriticalSection" @@ -54,7 +54,7 @@ } member { name: "Event" - mtype: "" + mtype: "" } member { name: "FIFOQueue" @@ -78,7 +78,7 @@ } member { name: "GPUOptions" - mtype: "" + mtype: "" } member { name: "GRAPH_DEF_VERSION" @@ -102,7 +102,7 @@ } member { name: "GraphDef" - mtype: "" + mtype: "" } member { name: "GraphKeys" @@ -110,11 +110,11 @@ } member { name: "GraphOptions" - mtype: "" + mtype: "" } member { name: "HistogramProto" - mtype: "" + mtype: "" } member { name: "IdentityReader" @@ -138,7 +138,7 @@ } member { name: "LogMessage" - mtype: "" + mtype: "" } member { name: "MONOLITHIC_BUILD" @@ -146,7 +146,7 @@ } member { name: "MetaGraphDef" - mtype: "" + mtype: "" } member { name: "Module" @@ -154,11 +154,11 @@ } member { name: "NameAttrList" - mtype: "" + mtype: "" } member { name: "NodeDef" - mtype: "" + mtype: "" } member { name: "OpError" @@ -170,7 +170,7 @@ } member { name: "OptimizerOptions" - mtype: "" + mtype: "" } member { name: "OptionalSpec" @@ -214,11 +214,11 @@ } member { name: "RunMetadata" - mtype: "" + mtype: "" } member { name: "RunOptions" - mtype: "" + mtype: "" } member { name: "Session" @@ -226,7 +226,7 @@ } member { name: "SessionLog" - mtype: "" + mtype: "" } member { name: "SparseConditionalAccumulator" @@ -250,11 +250,11 @@ } member { name: "Summary" - mtype: "" + mtype: "" } member { name: "SummaryMetadata" - mtype: "" + mtype: "" } member { name: "TFRecordReader" @@ -274,7 +274,7 @@ } member { name: "TensorInfo" - mtype: "" + mtype: "" } member { name: "TensorShape" E0310 20:36:05.805269 281472836531072 api_compatibility_test.py:393] None : --- +++ @@ -10,7 +10,7 @@ } member { name: "AttrValue" - mtype: "" + mtype: "" } member { name: "COMPILER_VERSION" @@ -34,7 +34,7 @@ } member { name: "ConfigProto" - mtype: "" + mtype: "" } member { name: "CriticalSection" @@ -54,7 +54,7 @@ } member { name: "Event" - mtype: "" + mtype: "" } member { name: "FIFOQueue" @@ -78,7 +78,7 @@ } member { name: "GPUOptions" - mtype: "" + mtype: "" } member { name: "GRAPH_DEF_VERSION" @@ -102,7 +102,7 @@ } member { name: "GraphDef" - mtype: "" + mtype: "" } member { name: "GraphKeys" @@ -110,11 +110,11 @@ } member { name: "GraphOptions" - mtype: "" + mtype: "" } member { name: "HistogramProto" - mtype: "" + mtype: "" } member { name: "IdentityReader" @@ -138,7 +138,7 @@ } member { name: "LogMessage" - mtype: "" + mtype: "" } member { name: "MONOLITHIC_BUILD" @@ -146,7 +146,7 @@ } member { name: "MetaGraphDef" - mtype: "" + mtype: "" } member { name: "Module" @@ -154,11 +154,11 @@ } member { name: "NameAttrList" - mtype: "" + mtype: "" } member { name: "NodeDef" - mtype: "" + mtype: "" } member { name: "OpError" @@ -170,7 +170,7 @@ } member { name: "OptimizerOptions" - mtype: "" + mtype: "" } member { name: "OptionalSpec" @@ -214,11 +214,11 @@ } member { name: "RunMetadata" - mtype: "" + mtype: "" } member { name: "RunOptions" - mtype: "" + mtype: "" } member { name: "Session" @@ -226,7 +226,7 @@ } member { name: "SessionLog" - mtype: "" + mtype: "" } member { name: "SparseConditionalAccumulator" @@ -250,11 +250,11 @@ } member { name: "Summary" - mtype: "" + mtype: "" } member { name: "SummaryMetadata" - mtype: "" + mtype: "" } member { name: "TFRecordReader" @@ -274,7 +274,7 @@ } member { name: "TensorInfo" - mtype: "" + mtype: "" } member { name: "TensorShape" ERROR:tensorflow: Change detected in python object: tensorflow.summary. E0310 20:36:05.805464 281472836531072 api_compatibility_test.py:392] Change detected in python object: tensorflow.summary. ERROR:tensorflow: path: "tensorflow.summary" tf_module { member { name: "Event" - mtype: "" + mtype: "" } member { name: "FileWriter" mtype: "" } member { name: "FileWriterCache" mtype: "" } member { name: "SessionLog" - mtype: "" + mtype: "" } member { name: "Summary" - mtype: "" + mtype: "" } member { name: "SummaryDescription" - mtype: "" + mtype: "" } member { name: "TaggedRunMetadata" - mtype: "" + mtype: "" } member_method { name: "all_v2_summary_ops" argspec: "args=[], varargs=None, keywords=None, defaults=None" } member_method { name: "audio" argspec: "args=[\'name\', \'tensor\', \'sample_rate\', \'max_outputs\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'3\', \'None\', \'None\'], " } member_method { name: "get_summary_description" argspec: "args=[\'node_def\'], varargs=None, keywords=None, defaults=None" } member_method { name: "histogram" argspec: "args=[\'name\', \'values\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "image" argspec: "args=[\'name\', \'tensor\', \'max_outputs\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'3\', \'None\', \'None\'], " } member_method { name: "initialize" argspec: "args=[\'graph\', \'session\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "merge" argspec: "args=[\'inputs\', \'collections\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "merge_all" argspec: "args=[\'key\', \'scope\', \'name\'], varargs=None, keywords=None, defaults=[\'summaries\', \'None\', \'None\'], " } member_method { name: "scalar" argspec: "args=[\'name\', \'tensor\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "tensor_summary" argspec: "args=[\'name\', \'tensor\', \'summary_description\', \'collections\', \'summary_metadata\', \'family\', \'display_name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'None\', \'None\'], " } member_method { name: "text" argspec: "args=[\'name\', \'tensor\', \'collections\'], varargs=None, keywords=None, defaults=[\'None\'], " } } E0310 20:36:05.805602 281472836531072 api_compatibility_test.py:393] path: "tensorflow.summary" tf_module { member { name: "Event" - mtype: "" + mtype: "" } member { name: "FileWriter" mtype: "" } member { name: "FileWriterCache" mtype: "" } member { name: "SessionLog" - mtype: "" + mtype: "" } member { name: "Summary" - mtype: "" + mtype: "" } member { name: "SummaryDescription" - mtype: "" + mtype: "" } member { name: "TaggedRunMetadata" - mtype: "" + mtype: "" } member_method { name: "all_v2_summary_ops" argspec: "args=[], varargs=None, keywords=None, defaults=None" } member_method { name: "audio" argspec: "args=[\'name\', \'tensor\', \'sample_rate\', \'max_outputs\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'3\', \'None\', \'None\'], " } member_method { name: "get_summary_description" argspec: "args=[\'node_def\'], varargs=None, keywords=None, defaults=None" } member_method { name: "histogram" argspec: "args=[\'name\', \'values\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "image" argspec: "args=[\'name\', \'tensor\', \'max_outputs\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'3\', \'None\', \'None\'], " } member_method { name: "initialize" argspec: "args=[\'graph\', \'session\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "merge" argspec: "args=[\'inputs\', \'collections\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "merge_all" argspec: "args=[\'key\', \'scope\', \'name\'], varargs=None, keywords=None, defaults=[\'summaries\', \'None\', \'None\'], " } member_method { name: "scalar" argspec: "args=[\'name\', \'tensor\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "tensor_summary" argspec: "args=[\'name\', \'tensor\', \'summary_description\', \'collections\', \'summary_metadata\', \'family\', \'display_name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'None\', \'None\'], " } member_method { name: "text" argspec: "args=[\'name\', \'tensor\', \'collections\'], varargs=None, keywords=None, defaults=[\'None\'], " } } INFO:tensorflow:time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibilityV1): 2.86s I0310 20:36:05.805843 281472836531072 test_util.py:2467] time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibilityV1): 2.86s [ FAILED ] ApiCompatibilityTest.testAPIBackwardsCompatibilityV1 [ RUN ] ApiCompatibilityTest.testAPIBackwardsCompatibilityV2 ERROR:tensorflow:TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. E0310 20:36:07.823042 281472836531072 api_compatibility_test.py:370] TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. ERROR:tensorflow:1 differences found between API and golden. E0310 20:36:07.823374 281472836531072 api_compatibility_test.py:371] 1 differences found between API and golden. ERROR:tensorflow: Change detected in python object: tensorflow.train. E0310 20:36:07.823535 281472836531072 api_compatibility_test.py:392] Change detected in python object: tensorflow.train. ERROR:tensorflow: path: "tensorflow.train" tf_module { member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointView" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "TrackableView" mtype: "" } member { name: "experimental" mtype: "" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } } E0310 20:36:07.823679 281472836531072 api_compatibility_test.py:393] path: "tensorflow.train" tf_module { member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointView" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "TrackableView" mtype: "" } member { name: "experimental" mtype: "" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } } INFO:tensorflow:time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibilityV2): 2.02s I0310 20:36:07.823933 281472836531072 test_util.py:2467] time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibilityV2): 2.02s [ FAILED ] ApiCompatibilityTest.testAPIBackwardsCompatibilityV2 [ RUN ] ApiCompatibilityTest.testNoSubclassOfMessage INFO:tensorflow:time(__main__.ApiCompatibilityTest.testNoSubclassOfMessage): 0.63s I0310 20:36:08.451378 281472836531072 test_util.py:2467] time(__main__.ApiCompatibilityTest.testNoSubclassOfMessage): 0.63s [ OK ] ApiCompatibilityTest.testNoSubclassOfMessage [ RUN ] ApiCompatibilityTest.testNoSubclassOfMessageV1 INFO:tensorflow:time(__main__.ApiCompatibilityTest.testNoSubclassOfMessageV1): 0.28s I0310 20:36:08.729813 281472836531072 test_util.py:2467] time(__main__.ApiCompatibilityTest.testNoSubclassOfMessageV1): 0.28s [ OK ] ApiCompatibilityTest.testNoSubclassOfMessageV1 [ RUN ] ApiCompatibilityTest.testNoSubclassOfMessageV2 INFO:tensorflow:time(__main__.ApiCompatibilityTest.testNoSubclassOfMessageV2): 0.29s I0310 20:36:09.024675 281472836531072 test_util.py:2467] time(__main__.ApiCompatibilityTest.testNoSubclassOfMessageV2): 0.29s [ OK ] ApiCompatibilityTest.testNoSubclassOfMessageV2 [ RUN ] ApiCompatibilityTest.test_session [ SKIPPED ] ApiCompatibilityTest.test_session ====================================================================== FAIL: testAPIBackwardsCompatibility (__main__.ApiCompatibilityTest) ApiCompatibilityTest.testAPIBackwardsCompatibility ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 511, in testAPIBackwardsCompatibility self._checkBackwardsCompatibility( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 488, in _checkBackwardsCompatibility self._AssertProtoDictEquals( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 395, in _AssertProtoDictEquals self.fail('%d differences found between API and golden.' % diff_count) AssertionError: 1 differences found between API and golden. ====================================================================== FAIL: testAPIBackwardsCompatibilityV1 (__main__.ApiCompatibilityTest) ApiCompatibilityTest.testAPIBackwardsCompatibilityV1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 528, in testAPIBackwardsCompatibilityV1 self._checkBackwardsCompatibility( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 488, in _checkBackwardsCompatibility self._AssertProtoDictEquals( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 395, in _AssertProtoDictEquals self.fail('%d differences found between API and golden.' % diff_count) AssertionError: 4 differences found between API and golden. ====================================================================== FAIL: testAPIBackwardsCompatibilityV2 (__main__.ApiCompatibilityTest) ApiCompatibilityTest.testAPIBackwardsCompatibilityV2 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 550, in testAPIBackwardsCompatibilityV2 self._checkBackwardsCompatibility( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 488, in _checkBackwardsCompatibility self._AssertProtoDictEquals( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 395, in _AssertProtoDictEquals self.fail('%d differences found between API and golden.' % diff_count) AssertionError: 1 differences found between API and golden. ---------------------------------------------------------------------- Ran 7 tests in 8.512s FAILED (failures=3, skipped=1) ================================================================================ ==================== Test output for //tensorflow/tools/api/tests:api_compatibility_test: Running tests under Python 3.9.16: /usr/local/bin/python3 [ RUN ] ApiCompatibilityTest.testAPIBackwardsCompatibility ERROR:tensorflow:TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. E0310 20:36:16.251858 281473270969216 api_compatibility_test.py:370] TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. ERROR:tensorflow:1 differences found between API and golden. E0310 20:36:16.252207 281473270969216 api_compatibility_test.py:371] 1 differences found between API and golden. ERROR:tensorflow: Change detected in python object: tensorflow.train. E0310 20:36:16.252362 281473270969216 api_compatibility_test.py:392] Change detected in python object: tensorflow.train. ERROR:tensorflow: path: "tensorflow.train" tf_module { member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointView" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "TrackableView" mtype: "" } member { name: "experimental" mtype: "" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } } E0310 20:36:16.252500 281473270969216 api_compatibility_test.py:393] path: "tensorflow.train" tf_module { member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointView" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "TrackableView" mtype: "" } member { name: "experimental" mtype: "" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } } INFO:tensorflow:time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibility): 2.24s I0310 20:36:16.252767 281473270969216 test_util.py:2467] time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibility): 2.24s [ FAILED ] ApiCompatibilityTest.testAPIBackwardsCompatibility [ RUN ] ApiCompatibilityTest.testAPIBackwardsCompatibilityV1 WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/python/util/decorator_utils.py:153: GraphKeys.VARIABLES (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.GraphKeys.GLOBAL_VARIABLES` instead. W0310 20:36:16.345221 281473270969216 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/python/util/decorator_utils.py:153: GraphKeys.VARIABLES (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.GraphKeys.GLOBAL_VARIABLES` instead. ERROR:tensorflow:TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. E0310 20:36:18.878980 281473270969216 api_compatibility_test.py:370] TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. ERROR:tensorflow:4 differences found between API and golden. E0310 20:36:18.879364 281473270969216 api_compatibility_test.py:371] 4 differences found between API and golden. ERROR:tensorflow: Change detected in python object: tensorflow. E0310 20:36:18.879519 281473270969216 api_compatibility_test.py:392] Change detected in python object: tensorflow. ERROR:tensorflow: None : --- +++ @@ -10,7 +10,7 @@ } member { name: "AttrValue" - mtype: "" + mtype: "" } member { name: "COMPILER_VERSION" @@ -34,7 +34,7 @@ } member { name: "ConfigProto" - mtype: "" + mtype: "" } member { name: "CriticalSection" @@ -54,7 +54,7 @@ } member { name: "Event" - mtype: "" + mtype: "" } member { name: "FIFOQueue" @@ -78,7 +78,7 @@ } member { name: "GPUOptions" - mtype: "" + mtype: "" } member { name: "GRAPH_DEF_VERSION" @@ -102,7 +102,7 @@ } member { name: "GraphDef" - mtype: "" + mtype: "" } member { name: "GraphKeys" @@ -110,11 +110,11 @@ } member { name: "GraphOptions" - mtype: "" + mtype: "" } member { name: "HistogramProto" - mtype: "" + mtype: "" } member { name: "IdentityReader" @@ -138,7 +138,7 @@ } member { name: "LogMessage" - mtype: "" + mtype: "" } member { name: "MONOLITHIC_BUILD" @@ -146,7 +146,7 @@ } member { name: "MetaGraphDef" - mtype: "" + mtype: "" } member { name: "Module" @@ -154,11 +154,11 @@ } member { name: "NameAttrList" - mtype: "" + mtype: "" } member { name: "NodeDef" - mtype: "" + mtype: "" } member { name: "OpError" @@ -170,7 +170,7 @@ } member { name: "OptimizerOptions" - mtype: "" + mtype: "" } member { name: "OptionalSpec" @@ -214,11 +214,11 @@ } member { name: "RunMetadata" - mtype: "" + mtype: "" } member { name: "RunOptions" - mtype: "" + mtype: "" } member { name: "Session" @@ -226,7 +226,7 @@ } member { name: "SessionLog" - mtype: "" + mtype: "" } member { name: "SparseConditionalAccumulator" @@ -250,11 +250,11 @@ } member { name: "Summary" - mtype: "" + mtype: "" } member { name: "SummaryMetadata" - mtype: "" + mtype: "" } member { name: "TFRecordReader" @@ -274,7 +274,7 @@ } member { name: "TensorInfo" - mtype: "" + mtype: "" } member { name: "TensorShape" E0310 20:36:18.879657 281473270969216 api_compatibility_test.py:393] None : --- +++ @@ -10,7 +10,7 @@ } member { name: "AttrValue" - mtype: "" + mtype: "" } member { name: "COMPILER_VERSION" @@ -34,7 +34,7 @@ } member { name: "ConfigProto" - mtype: "" + mtype: "" } member { name: "CriticalSection" @@ -54,7 +54,7 @@ } member { name: "Event" - mtype: "" + mtype: "" } member { name: "FIFOQueue" @@ -78,7 +78,7 @@ } member { name: "GPUOptions" - mtype: "" + mtype: "" } member { name: "GRAPH_DEF_VERSION" @@ -102,7 +102,7 @@ } member { name: "GraphDef" - mtype: "" + mtype: "" } member { name: "GraphKeys" @@ -110,11 +110,11 @@ } member { name: "GraphOptions" - mtype: "" + mtype: "" } member { name: "HistogramProto" - mtype: "" + mtype: "" } member { name: "IdentityReader" @@ -138,7 +138,7 @@ } member { name: "LogMessage" - mtype: "" + mtype: "" } member { name: "MONOLITHIC_BUILD" @@ -146,7 +146,7 @@ } member { name: "MetaGraphDef" - mtype: "" + mtype: "" } member { name: "Module" @@ -154,11 +154,11 @@ } member { name: "NameAttrList" - mtype: "" + mtype: "" } member { name: "NodeDef" - mtype: "" + mtype: "" } member { name: "OpError" @@ -170,7 +170,7 @@ } member { name: "OptimizerOptions" - mtype: "" + mtype: "" } member { name: "OptionalSpec" @@ -214,11 +214,11 @@ } member { name: "RunMetadata" - mtype: "" + mtype: "" } member { name: "RunOptions" - mtype: "" + mtype: "" } member { name: "Session" @@ -226,7 +226,7 @@ } member { name: "SessionLog" - mtype: "" + mtype: "" } member { name: "SparseConditionalAccumulator" @@ -250,11 +250,11 @@ } member { name: "Summary" - mtype: "" + mtype: "" } member { name: "SummaryMetadata" - mtype: "" + mtype: "" } member { name: "TFRecordReader" @@ -274,7 +274,7 @@ } member { name: "TensorInfo" - mtype: "" + mtype: "" } member { name: "TensorShape" ERROR:tensorflow: Change detected in python object: tensorflow.profiler. E0310 20:36:18.879816 281473270969216 api_compatibility_test.py:392] Change detected in python object: tensorflow.profiler. ERROR:tensorflow: path: "tensorflow.profiler" tf_module { member { name: "AdviceProto" - mtype: "" + mtype: "" } member { name: "GraphNodeProto" - mtype: "" + mtype: "" } member { name: "MultiGraphNodeProto" - mtype: "" + mtype: "" } member { name: "OpLogProto" - mtype: "" + mtype: "" } member { name: "ProfileOptionBuilder" mtype: "" } member { name: "Profiler" mtype: "" } member_method { name: "advise" argspec: "args=[\'graph\', \'run_meta\', \'options\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'0\'], " } member_method { name: "profile" argspec: "args=[\'graph\', \'run_meta\', \'op_log\', \'cmd\', \'options\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'scope\', \'0\'], " } member_method { name: "write_op_log" argspec: "args=[\'graph\', \'log_dir\', \'op_log\', \'run_meta\', \'add_trace\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\'], " } } E0310 20:36:18.879950 281473270969216 api_compatibility_test.py:393] path: "tensorflow.profiler" tf_module { member { name: "AdviceProto" - mtype: "" + mtype: "" } member { name: "GraphNodeProto" - mtype: "" + mtype: "" } member { name: "MultiGraphNodeProto" - mtype: "" + mtype: "" } member { name: "OpLogProto" - mtype: "" + mtype: "" } member { name: "ProfileOptionBuilder" mtype: "" } member { name: "Profiler" mtype: "" } member_method { name: "advise" argspec: "args=[\'graph\', \'run_meta\', \'options\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'0\'], " } member_method { name: "profile" argspec: "args=[\'graph\', \'run_meta\', \'op_log\', \'cmd\', \'options\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'scope\', \'0\'], " } member_method { name: "write_op_log" argspec: "args=[\'graph\', \'log_dir\', \'op_log\', \'run_meta\', \'add_trace\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\'], " } } ERROR:tensorflow: Change detected in python object: tensorflow.train. E0310 20:36:18.880095 281473270969216 api_compatibility_test.py:392] Change detected in python object: tensorflow.train. ERROR:tensorflow: path: "tensorflow.train" tf_module { member { name: "AdadeltaOptimizer" mtype: "" } member { name: "AdagradDAOptimizer" mtype: "" } member { name: "AdagradOptimizer" mtype: "" } member { name: "AdamOptimizer" mtype: "" } member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointSaverHook" mtype: "" } member { name: "CheckpointSaverListener" mtype: "" } member { name: "ChiefSessionCreator" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FeedFnHook" mtype: "" } member { name: "FinalOpsHook" mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "FtrlOptimizer" mtype: "" } member { name: "GlobalStepWaiterHook" mtype: "" } member { name: "GradientDescentOptimizer" mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "LoggingTensorHook" mtype: "" } member { name: "LooperThread" mtype: "" } member { name: "MomentumOptimizer" mtype: "" } member { name: "MonitoredSession" mtype: "" } member { name: "NanLossDuringTrainingError" mtype: "" } member { name: "NanTensorHook" mtype: "" } member { name: "Optimizer" mtype: "" } member { name: "ProfilerHook" mtype: "" } member { name: "ProximalAdagradOptimizer" mtype: "" } member { name: "ProximalGradientDescentOptimizer" mtype: "" } member { name: "QueueRunner" mtype: "" } member { name: "RMSPropOptimizer" mtype: "" } member { name: "Saver" mtype: "" } member { name: "SaverDef" - mtype: "" + mtype: "" } member { name: "Scaffold" mtype: "" } member { name: "SecondOrStepTimer" mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "Server" mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "SessionCreator" mtype: "" } member { name: "SessionManager" mtype: "" } member { name: "SessionRunArgs" mtype: "" } member { name: "SessionRunContext" mtype: "" } member { name: "SessionRunHook" mtype: "" } member { name: "SessionRunValues" mtype: "" } member { name: "SingularMonitoredSession" mtype: "" } member { name: "StepCounterHook" mtype: "" } member { name: "StopAtStepHook" mtype: "" } member { name: "SummarySaverHook" mtype: "" } member { name: "Supervisor" mtype: "" } member { name: "SyncReplicasOptimizer" mtype: "" } member { name: "VocabInfo" mtype: "" } member { name: "WorkerSessionCreator" mtype: "" } member { name: "experimental" mtype: "" } member { name: "queue_runner" mtype: "" } member_method { name: "MonitoredTrainingSession" argspec: "args=[\'master\', \'is_chief\', \'checkpoint_dir\', \'scaffold\', \'hooks\', \'chief_only_hooks\', \'save_checkpoint_secs\', \'save_summaries_steps\', \'save_summaries_secs\', \'config\', \'stop_grace_period_secs\', \'log_step_count_steps\', \'max_wait_secs\', \'save_checkpoint_steps\', \'summary_dir\', \'save_graph_def\'], varargs=None, keywords=None, defaults=[\'\', \'True\', \'None\', \'None\', \'None\', \'None\', \'\', \'\', \'\', \'None\', \'120\', \'100\', \'7200\', \'\', \'None\', \'True\'], " } member_method { name: "NewCheckpointReader" argspec: "args=[\'filepattern\'], varargs=None, keywords=None, defaults=None" } member_method { name: "add_queue_runner" argspec: "args=[\'qr\', \'collection\'], varargs=None, keywords=None, defaults=[\'queue_runners\'], " } member_method { name: "assert_global_step" argspec: "args=[\'global_step_tensor\'], varargs=None, keywords=None, defaults=None" } member_method { name: "basic_train_loop" argspec: "args=[\'supervisor\', \'train_step_fn\', \'args\', \'kwargs\', \'master\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'\'], " } member_method { name: "batch" argspec: "args=[\'tensors\', \'batch_size\', \'num_threads\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "checkpoint_exists" argspec: "args=[\'checkpoint_prefix\'], varargs=None, keywords=None, defaults=None" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'alpha\', \'name\'], varargs=None, keywords=None, defaults=[\'0.0\', \'None\'], " } member_method { name: "cosine_decay_restarts" argspec: "args=[\'learning_rate\', \'global_step\', \'first_decay_steps\', \'t_mul\', \'m_mul\', \'alpha\', \'name\'], varargs=None, keywords=None, defaults=[\'2.0\', \'1.0\', \'0.0\', \'None\'], " } member_method { name: "create_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "do_quantize_training_on_graphdef" argspec: "args=[\'input_graph\', \'num_bits\'], varargs=None, keywords=None, defaults=None" } member_method { name: "exponential_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "export_meta_graph" argspec: "args=[\'filename\', \'meta_info_def\', \'graph_def\', \'saver_def\', \'collection_list\', \'as_text\', \'graph\', \'export_scope\', \'clear_devices\', \'clear_extraneous_savers\', \'strip_default_attrs\', \'save_debug_info\'], varargs=None, keywords=kwargs, defaults=[\'None\', \'None\', \'None\', \'None\', \'None\', \'False\', \'None\', \'None\', \'False\', \'False\', \'False\', \'False\'], " } member_method { name: "generate_checkpoint_state_proto" argspec: "args=[\'save_dir\', \'model_checkpoint_path\', \'all_model_checkpoint_paths\', \'all_model_checkpoint_timestamps\', \'last_preserved_timestamp\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_mtimes" argspec: "args=[\'checkpoint_prefixes\'], varargs=None, keywords=None, defaults=None" } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "get_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "get_or_create_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "global_step" argspec: "args=[\'sess\', \'global_step_tensor\'], varargs=None, keywords=None, defaults=None" } member_method { name: "import_meta_graph" argspec: "args=[\'meta_graph_or_file\', \'clear_devices\', \'import_scope\'], varargs=None, keywords=kwargs, defaults=[\'False\', \'None\'], " } member_method { name: "init_from_checkpoint" argspec: "args=[\'ckpt_dir_or_file\', \'assignment_map\'], varargs=None, keywords=None, defaults=None" } member_method { name: "input_producer" argspec: "args=[\'input_tensor\', \'element_shape\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'summary_name\', \'name\', \'cancel_op\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\', \'None\', \'32\', \'None\', \'None\', \'None\', \'None\'], " } member_method { name: "inverse_time_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "limit_epochs" argspec: "args=[\'tensor\', \'num_epochs\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "linear_cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'num_periods\', \'alpha\', \'beta\', \'name\'], varargs=None, keywords=None, defaults=[\'0.5\', \'0.0\', \'0.001\', \'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } member_method { name: "match_filenames_once" argspec: "args=[\'pattern\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "maybe_batch" argspec: "args=[\'tensors\', \'keep_input\', \'batch_size\', \'num_threads\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_batch_join" argspec: "args=[\'tensors_list\', \'keep_input\', \'batch_size\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_shuffle_batch" argspec: "args=[\'tensors\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'keep_input\', \'num_threads\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_shuffle_batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'keep_input\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "natural_exp_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "noisy_linear_cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'initial_variance\', \'variance_decay\', \'num_periods\', \'alpha\', \'beta\', \'name\'], varargs=None, keywords=None, defaults=[\'1.0\', \'0.55\', \'0.5\', \'0.0\', \'0.001\', \'None\'], " } member_method { name: "piecewise_constant" argspec: "args=[\'x\', \'boundaries\', \'values\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "piecewise_constant_decay" argspec: "args=[\'x\', \'boundaries\', \'values\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "polynomial_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'end_learning_rate\', \'power\', \'cycle\', \'name\'], varargs=None, keywords=None, defaults=[\'0.0001\', \'1.0\', \'False\', \'None\'], " } member_method { name: "range_input_producer" argspec: "args=[\'limit\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\'], " } member_method { name: "remove_checkpoint" argspec: "args=[\'checkpoint_prefix\', \'checkpoint_format_version\', \'meta_graph_suffix\'], varargs=None, keywords=None, defaults=[\'2\', \'meta\'], " } member_method { name: "replica_device_setter" argspec: "args=[\'ps_tasks\', \'ps_device\', \'worker_device\', \'merge_devices\', \'cluster\', \'ps_ops\', \'ps_strategy\'], varargs=None, keywords=None, defaults=[\'0\', \'/job:ps\', \'/job:worker\', \'True\', \'None\', \'None\', \'None\'], " } member_method { name: "sdca_fprint" argspec: "args=[\'input\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "sdca_optimizer" argspec: "args=[\'sparse_example_indices\', \'sparse_feature_indices\', \'sparse_feature_values\', \'dense_features\', \'example_weights\', \'example_labels\', \'sparse_indices\', \'sparse_weights\', \'dense_weights\', \'example_state_data\', \'loss_type\', \'l1\', \'l2\', \'num_loss_partitions\', \'num_inner_iterations\', \'adaptative\', \'name\'], varargs=None, keywords=None, defaults=[\'True\', \'None\'], " } member_method { name: "sdca_shrink_l1" argspec: "args=[\'weights\', \'l1\', \'l2\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "shuffle_batch" argspec: "args=[\'tensors\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'num_threads\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "shuffle_batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "slice_input_producer" argspec: "args=[\'tensor_list\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\'], " } member_method { name: "start_queue_runners" argspec: "args=[\'sess\', \'coord\', \'daemon\', \'start\', \'collection\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\', \'True\', \'queue_runners\'], " } member_method { name: "string_input_producer" argspec: "args=[\'string_tensor\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\', \'cancel_op\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\', \'None\'], " } member_method { name: "summary_iterator" argspec: "args=[\'path\'], varargs=None, keywords=None, defaults=None" } member_method { name: "update_checkpoint_state" argspec: "args=[\'save_dir\', \'model_checkpoint_path\', \'all_model_checkpoint_paths\', \'latest_filename\', \'all_model_checkpoint_timestamps\', \'last_preserved_timestamp\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'None\'], " } member_method { name: "warm_start" argspec: "args=[\'ckpt_to_initialize_from\', \'vars_to_warm_start\', \'var_name_to_vocab_info\', \'var_name_to_prev_var_name\'], varargs=None, keywords=None, defaults=[\'.*\', \'None\', \'None\'], " } member_method { name: "write_graph" argspec: "args=[\'graph_or_graph_def\', \'logdir\', \'name\', \'as_text\'], varargs=None, keywords=None, defaults=[\'True\'], " } } E0310 20:36:18.880225 281473270969216 api_compatibility_test.py:393] path: "tensorflow.train" tf_module { member { name: "AdadeltaOptimizer" mtype: "" } member { name: "AdagradDAOptimizer" mtype: "" } member { name: "AdagradOptimizer" mtype: "" } member { name: "AdamOptimizer" mtype: "" } member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointSaverHook" mtype: "" } member { name: "CheckpointSaverListener" mtype: "" } member { name: "ChiefSessionCreator" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FeedFnHook" mtype: "" } member { name: "FinalOpsHook" mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "FtrlOptimizer" mtype: "" } member { name: "GlobalStepWaiterHook" mtype: "" } member { name: "GradientDescentOptimizer" mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "LoggingTensorHook" mtype: "" } member { name: "LooperThread" mtype: "" } member { name: "MomentumOptimizer" mtype: "" } member { name: "MonitoredSession" mtype: "" } member { name: "NanLossDuringTrainingError" mtype: "" } member { name: "NanTensorHook" mtype: "" } member { name: "Optimizer" mtype: "" } member { name: "ProfilerHook" mtype: "" } member { name: "ProximalAdagradOptimizer" mtype: "" } member { name: "ProximalGradientDescentOptimizer" mtype: "" } member { name: "QueueRunner" mtype: "" } member { name: "RMSPropOptimizer" mtype: "" } member { name: "Saver" mtype: "" } member { name: "SaverDef" - mtype: "" + mtype: "" } member { name: "Scaffold" mtype: "" } member { name: "SecondOrStepTimer" mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "Server" mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "SessionCreator" mtype: "" } member { name: "SessionManager" mtype: "" } member { name: "SessionRunArgs" mtype: "" } member { name: "SessionRunContext" mtype: "" } member { name: "SessionRunHook" mtype: "" } member { name: "SessionRunValues" mtype: "" } member { name: "SingularMonitoredSession" mtype: "" } member { name: "StepCounterHook" mtype: "" } member { name: "StopAtStepHook" mtype: "" } member { name: "SummarySaverHook" mtype: "" } member { name: "Supervisor" mtype: "" } member { name: "SyncReplicasOptimizer" mtype: "" } member { name: "VocabInfo" mtype: "" } member { name: "WorkerSessionCreator" mtype: "" } member { name: "experimental" mtype: "" } member { name: "queue_runner" mtype: "" } member_method { name: "MonitoredTrainingSession" argspec: "args=[\'master\', \'is_chief\', \'checkpoint_dir\', \'scaffold\', \'hooks\', \'chief_only_hooks\', \'save_checkpoint_secs\', \'save_summaries_steps\', \'save_summaries_secs\', \'config\', \'stop_grace_period_secs\', \'log_step_count_steps\', \'max_wait_secs\', \'save_checkpoint_steps\', \'summary_dir\', \'save_graph_def\'], varargs=None, keywords=None, defaults=[\'\', \'True\', \'None\', \'None\', \'None\', \'None\', \'\', \'\', \'\', \'None\', \'120\', \'100\', \'7200\', \'\', \'None\', \'True\'], " } member_method { name: "NewCheckpointReader" argspec: "args=[\'filepattern\'], varargs=None, keywords=None, defaults=None" } member_method { name: "add_queue_runner" argspec: "args=[\'qr\', \'collection\'], varargs=None, keywords=None, defaults=[\'queue_runners\'], " } member_method { name: "assert_global_step" argspec: "args=[\'global_step_tensor\'], varargs=None, keywords=None, defaults=None" } member_method { name: "basic_train_loop" argspec: "args=[\'supervisor\', \'train_step_fn\', \'args\', \'kwargs\', \'master\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'\'], " } member_method { name: "batch" argspec: "args=[\'tensors\', \'batch_size\', \'num_threads\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "checkpoint_exists" argspec: "args=[\'checkpoint_prefix\'], varargs=None, keywords=None, defaults=None" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'alpha\', \'name\'], varargs=None, keywords=None, defaults=[\'0.0\', \'None\'], " } member_method { name: "cosine_decay_restarts" argspec: "args=[\'learning_rate\', \'global_step\', \'first_decay_steps\', \'t_mul\', \'m_mul\', \'alpha\', \'name\'], varargs=None, keywords=None, defaults=[\'2.0\', \'1.0\', \'0.0\', \'None\'], " } member_method { name: "create_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "do_quantize_training_on_graphdef" argspec: "args=[\'input_graph\', \'num_bits\'], varargs=None, keywords=None, defaults=None" } member_method { name: "exponential_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "export_meta_graph" argspec: "args=[\'filename\', \'meta_info_def\', \'graph_def\', \'saver_def\', \'collection_list\', \'as_text\', \'graph\', \'export_scope\', \'clear_devices\', \'clear_extraneous_savers\', \'strip_default_attrs\', \'save_debug_info\'], varargs=None, keywords=kwargs, defaults=[\'None\', \'None\', \'None\', \'None\', \'None\', \'False\', \'None\', \'None\', \'False\', \'False\', \'False\', \'False\'], " } member_method { name: "generate_checkpoint_state_proto" argspec: "args=[\'save_dir\', \'model_checkpoint_path\', \'all_model_checkpoint_paths\', \'all_model_checkpoint_timestamps\', \'last_preserved_timestamp\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_mtimes" argspec: "args=[\'checkpoint_prefixes\'], varargs=None, keywords=None, defaults=None" } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "get_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "get_or_create_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "global_step" argspec: "args=[\'sess\', \'global_step_tensor\'], varargs=None, keywords=None, defaults=None" } member_method { name: "import_meta_graph" argspec: "args=[\'meta_graph_or_file\', \'clear_devices\', \'import_scope\'], varargs=None, keywords=kwargs, defaults=[\'False\', \'None\'], " } member_method { name: "init_from_checkpoint" argspec: "args=[\'ckpt_dir_or_file\', \'assignment_map\'], varargs=None, keywords=None, defaults=None" } member_method { name: "input_producer" argspec: "args=[\'input_tensor\', \'element_shape\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'summary_name\', \'name\', \'cancel_op\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\', \'None\', \'32\', \'None\', \'None\', \'None\', \'None\'], " } member_method { name: "inverse_time_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "limit_epochs" argspec: "args=[\'tensor\', \'num_epochs\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "linear_cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'num_periods\', \'alpha\', \'beta\', \'name\'], varargs=None, keywords=None, defaults=[\'0.5\', \'0.0\', \'0.001\', \'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } member_method { name: "match_filenames_once" argspec: "args=[\'pattern\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "maybe_batch" argspec: "args=[\'tensors\', \'keep_input\', \'batch_size\', \'num_threads\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_batch_join" argspec: "args=[\'tensors_list\', \'keep_input\', \'batch_size\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_shuffle_batch" argspec: "args=[\'tensors\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'keep_input\', \'num_threads\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_shuffle_batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'keep_input\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "natural_exp_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "noisy_linear_cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'initial_variance\', \'variance_decay\', \'num_periods\', \'alpha\', \'beta\', \'name\'], varargs=None, keywords=None, defaults=[\'1.0\', \'0.55\', \'0.5\', \'0.0\', \'0.001\', \'None\'], " } member_method { name: "piecewise_constant" argspec: "args=[\'x\', \'boundaries\', \'values\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "piecewise_constant_decay" argspec: "args=[\'x\', \'boundaries\', \'values\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "polynomial_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'end_learning_rate\', \'power\', \'cycle\', \'name\'], varargs=None, keywords=None, defaults=[\'0.0001\', \'1.0\', \'False\', \'None\'], " } member_method { name: "range_input_producer" argspec: "args=[\'limit\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\'], " } member_method { name: "remove_checkpoint" argspec: "args=[\'checkpoint_prefix\', \'checkpoint_format_version\', \'meta_graph_suffix\'], varargs=None, keywords=None, defaults=[\'2\', \'meta\'], " } member_method { name: "replica_device_setter" argspec: "args=[\'ps_tasks\', \'ps_device\', \'worker_device\', \'merge_devices\', \'cluster\', \'ps_ops\', \'ps_strategy\'], varargs=None, keywords=None, defaults=[\'0\', \'/job:ps\', \'/job:worker\', \'True\', \'None\', \'None\', \'None\'], " } member_method { name: "sdca_fprint" argspec: "args=[\'input\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "sdca_optimizer" argspec: "args=[\'sparse_example_indices\', \'sparse_feature_indices\', \'sparse_feature_values\', \'dense_features\', \'example_weights\', \'example_labels\', \'sparse_indices\', \'sparse_weights\', \'dense_weights\', \'example_state_data\', \'loss_type\', \'l1\', \'l2\', \'num_loss_partitions\', \'num_inner_iterations\', \'adaptative\', \'name\'], varargs=None, keywords=None, defaults=[\'True\', \'None\'], " } member_method { name: "sdca_shrink_l1" argspec: "args=[\'weights\', \'l1\', \'l2\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "shuffle_batch" argspec: "args=[\'tensors\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'num_threads\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "shuffle_batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "slice_input_producer" argspec: "args=[\'tensor_list\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\'], " } member_method { name: "start_queue_runners" argspec: "args=[\'sess\', \'coord\', \'daemon\', \'start\', \'collection\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\', \'True\', \'queue_runners\'], " } member_method { name: "string_input_producer" argspec: "args=[\'string_tensor\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\', \'cancel_op\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\', \'None\'], " } member_method { name: "summary_iterator" argspec: "args=[\'path\'], varargs=None, keywords=None, defaults=None" } member_method { name: "update_checkpoint_state" argspec: "args=[\'save_dir\', \'model_checkpoint_path\', \'all_model_checkpoint_paths\', \'latest_filename\', \'all_model_checkpoint_timestamps\', \'last_preserved_timestamp\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'None\'], " } member_method { name: "warm_start" argspec: "args=[\'ckpt_to_initialize_from\', \'vars_to_warm_start\', \'var_name_to_vocab_info\', \'var_name_to_prev_var_name\'], varargs=None, keywords=None, defaults=[\'.*\', \'None\', \'None\'], " } member_method { name: "write_graph" argspec: "args=[\'graph_or_graph_def\', \'logdir\', \'name\', \'as_text\'], varargs=None, keywords=None, defaults=[\'True\'], " } } ERROR:tensorflow: Change detected in python object: tensorflow.summary. E0310 20:36:18.880431 281473270969216 api_compatibility_test.py:392] Change detected in python object: tensorflow.summary. ERROR:tensorflow: path: "tensorflow.summary" tf_module { member { name: "Event" - mtype: "" + mtype: "" } member { name: "FileWriter" mtype: "" } member { name: "FileWriterCache" mtype: "" } member { name: "SessionLog" - mtype: "" + mtype: "" } member { name: "Summary" - mtype: "" + mtype: "" } member { name: "SummaryDescription" - mtype: "" + mtype: "" } member { name: "TaggedRunMetadata" - mtype: "" + mtype: "" } member_method { name: "all_v2_summary_ops" argspec: "args=[], varargs=None, keywords=None, defaults=None" } member_method { name: "audio" argspec: "args=[\'name\', \'tensor\', \'sample_rate\', \'max_outputs\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'3\', \'None\', \'None\'], " } member_method { name: "get_summary_description" argspec: "args=[\'node_def\'], varargs=None, keywords=None, defaults=None" } member_method { name: "histogram" argspec: "args=[\'name\', \'values\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "image" argspec: "args=[\'name\', \'tensor\', \'max_outputs\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'3\', \'None\', \'None\'], " } member_method { name: "initialize" argspec: "args=[\'graph\', \'session\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "merge" argspec: "args=[\'inputs\', \'collections\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "merge_all" argspec: "args=[\'key\', \'scope\', \'name\'], varargs=None, keywords=None, defaults=[\'summaries\', \'None\', \'None\'], " } member_method { name: "scalar" argspec: "args=[\'name\', \'tensor\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "tensor_summary" argspec: "args=[\'name\', \'tensor\', \'summary_description\', \'collections\', \'summary_metadata\', \'family\', \'display_name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'None\', \'None\'], " } member_method { name: "text" argspec: "args=[\'name\', \'tensor\', \'collections\'], varargs=None, keywords=None, defaults=[\'None\'], " } } E0310 20:36:18.880564 281473270969216 api_compatibility_test.py:393] path: "tensorflow.summary" tf_module { member { name: "Event" - mtype: "" + mtype: "" } member { name: "FileWriter" mtype: "" } member { name: "FileWriterCache" mtype: "" } member { name: "SessionLog" - mtype: "" + mtype: "" } member { name: "Summary" - mtype: "" + mtype: "" } member { name: "SummaryDescription" - mtype: "" + mtype: "" } member { name: "TaggedRunMetadata" - mtype: "" + mtype: "" } member_method { name: "all_v2_summary_ops" argspec: "args=[], varargs=None, keywords=None, defaults=None" } member_method { name: "audio" argspec: "args=[\'name\', \'tensor\', \'sample_rate\', \'max_outputs\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'3\', \'None\', \'None\'], " } member_method { name: "get_summary_description" argspec: "args=[\'node_def\'], varargs=None, keywords=None, defaults=None" } member_method { name: "histogram" argspec: "args=[\'name\', \'values\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "image" argspec: "args=[\'name\', \'tensor\', \'max_outputs\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'3\', \'None\', \'None\'], " } member_method { name: "initialize" argspec: "args=[\'graph\', \'session\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "merge" argspec: "args=[\'inputs\', \'collections\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "merge_all" argspec: "args=[\'key\', \'scope\', \'name\'], varargs=None, keywords=None, defaults=[\'summaries\', \'None\', \'None\'], " } member_method { name: "scalar" argspec: "args=[\'name\', \'tensor\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "tensor_summary" argspec: "args=[\'name\', \'tensor\', \'summary_description\', \'collections\', \'summary_metadata\', \'family\', \'display_name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'None\', \'None\'], " } member_method { name: "text" argspec: "args=[\'name\', \'tensor\', \'collections\'], varargs=None, keywords=None, defaults=[\'None\'], " } } INFO:tensorflow:time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibilityV1): 2.63s I0310 20:36:18.880825 281473270969216 test_util.py:2467] time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibilityV1): 2.63s [ FAILED ] ApiCompatibilityTest.testAPIBackwardsCompatibilityV1 [ RUN ] ApiCompatibilityTest.testAPIBackwardsCompatibilityV2 ERROR:tensorflow:TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. E0310 20:36:20.870069 281473270969216 api_compatibility_test.py:370] TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. ERROR:tensorflow:1 differences found between API and golden. E0310 20:36:20.870365 281473270969216 api_compatibility_test.py:371] 1 differences found between API and golden. ERROR:tensorflow: Change detected in python object: tensorflow.train. E0310 20:36:20.870517 281473270969216 api_compatibility_test.py:392] Change detected in python object: tensorflow.train. ERROR:tensorflow: path: "tensorflow.train" tf_module { member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointView" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "TrackableView" mtype: "" } member { name: "experimental" mtype: "" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } } E0310 20:36:20.870657 281473270969216 api_compatibility_test.py:393] path: "tensorflow.train" tf_module { member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointView" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "TrackableView" mtype: "" } member { name: "experimental" mtype: "" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } } INFO:tensorflow:time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibilityV2): 1.99s I0310 20:36:20.870891 281473270969216 test_util.py:2467] time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibilityV2): 1.99s [ FAILED ] ApiCompatibilityTest.testAPIBackwardsCompatibilityV2 [ RUN ] ApiCompatibilityTest.testNoSubclassOfMessage INFO:tensorflow:time(__main__.ApiCompatibilityTest.testNoSubclassOfMessage): 0.64s I0310 20:36:21.509473 281473270969216 test_util.py:2467] time(__main__.ApiCompatibilityTest.testNoSubclassOfMessage): 0.64s [ OK ] ApiCompatibilityTest.testNoSubclassOfMessage [ RUN ] ApiCompatibilityTest.testNoSubclassOfMessageV1 INFO:tensorflow:time(__main__.ApiCompatibilityTest.testNoSubclassOfMessageV1): 0.28s I0310 20:36:21.794312 281473270969216 test_util.py:2467] time(__main__.ApiCompatibilityTest.testNoSubclassOfMessageV1): 0.28s [ OK ] ApiCompatibilityTest.testNoSubclassOfMessageV1 [ RUN ] ApiCompatibilityTest.testNoSubclassOfMessageV2 INFO:tensorflow:time(__main__.ApiCompatibilityTest.testNoSubclassOfMessageV2): 0.3s I0310 20:36:22.094392 281473270969216 test_util.py:2467] time(__main__.ApiCompatibilityTest.testNoSubclassOfMessageV2): 0.3s [ OK ] ApiCompatibilityTest.testNoSubclassOfMessageV2 [ RUN ] ApiCompatibilityTest.test_session [ SKIPPED ] ApiCompatibilityTest.test_session ====================================================================== FAIL: testAPIBackwardsCompatibility (__main__.ApiCompatibilityTest) ApiCompatibilityTest.testAPIBackwardsCompatibility ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 511, in testAPIBackwardsCompatibility self._checkBackwardsCompatibility( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 488, in _checkBackwardsCompatibility self._AssertProtoDictEquals( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 395, in _AssertProtoDictEquals self.fail('%d differences found between API and golden.' % diff_count) AssertionError: 1 differences found between API and golden. ====================================================================== FAIL: testAPIBackwardsCompatibilityV1 (__main__.ApiCompatibilityTest) ApiCompatibilityTest.testAPIBackwardsCompatibilityV1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 528, in testAPIBackwardsCompatibilityV1 self._checkBackwardsCompatibility( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 488, in _checkBackwardsCompatibility self._AssertProtoDictEquals( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 395, in _AssertProtoDictEquals self.fail('%d differences found between API and golden.' % diff_count) AssertionError: 4 differences found between API and golden. ====================================================================== FAIL: testAPIBackwardsCompatibilityV2 (__main__.ApiCompatibilityTest) ApiCompatibilityTest.testAPIBackwardsCompatibilityV2 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 550, in testAPIBackwardsCompatibilityV2 self._checkBackwardsCompatibility( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 488, in _checkBackwardsCompatibility self._AssertProtoDictEquals( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 395, in _AssertProtoDictEquals self.fail('%d differences found between API and golden.' % diff_count) AssertionError: 1 differences found between API and golden. ---------------------------------------------------------------------- Ran 7 tests in 8.079s FAILED (failures=3, skipped=1) ================================================================================ ==================== Test output for //tensorflow/tools/api/tests:api_compatibility_test: Running tests under Python 3.9.16: /usr/local/bin/python3 [ RUN ] ApiCompatibilityTest.testAPIBackwardsCompatibility ERROR:tensorflow:TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. E0310 20:36:28.931105 281473543271296 api_compatibility_test.py:370] TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. ERROR:tensorflow:1 differences found between API and golden. E0310 20:36:28.931417 281473543271296 api_compatibility_test.py:371] 1 differences found between API and golden. ERROR:tensorflow: Change detected in python object: tensorflow.train. E0310 20:36:28.931572 281473543271296 api_compatibility_test.py:392] Change detected in python object: tensorflow.train. ERROR:tensorflow: path: "tensorflow.train" tf_module { member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointView" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "TrackableView" mtype: "" } member { name: "experimental" mtype: "" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } } E0310 20:36:28.931712 281473543271296 api_compatibility_test.py:393] path: "tensorflow.train" tf_module { member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointView" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "TrackableView" mtype: "" } member { name: "experimental" mtype: "" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } } INFO:tensorflow:time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibility): 2.23s I0310 20:36:28.931960 281473543271296 test_util.py:2467] time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibility): 2.23s [ FAILED ] ApiCompatibilityTest.testAPIBackwardsCompatibility [ RUN ] ApiCompatibilityTest.testAPIBackwardsCompatibilityV1 WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/python/util/decorator_utils.py:153: GraphKeys.VARIABLES (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.GraphKeys.GLOBAL_VARIABLES` instead. W0310 20:36:29.022679 281473543271296 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/python/util/decorator_utils.py:153: GraphKeys.VARIABLES (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.GraphKeys.GLOBAL_VARIABLES` instead. ERROR:tensorflow:TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. E0310 20:36:31.541967 281473543271296 api_compatibility_test.py:370] TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. ERROR:tensorflow:4 differences found between API and golden. E0310 20:36:31.542290 281473543271296 api_compatibility_test.py:371] 4 differences found between API and golden. ERROR:tensorflow: Change detected in python object: tensorflow.train. E0310 20:36:31.542446 281473543271296 api_compatibility_test.py:392] Change detected in python object: tensorflow.train. ERROR:tensorflow: path: "tensorflow.train" tf_module { member { name: "AdadeltaOptimizer" mtype: "" } member { name: "AdagradDAOptimizer" mtype: "" } member { name: "AdagradOptimizer" mtype: "" } member { name: "AdamOptimizer" mtype: "" } member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointSaverHook" mtype: "" } member { name: "CheckpointSaverListener" mtype: "" } member { name: "ChiefSessionCreator" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FeedFnHook" mtype: "" } member { name: "FinalOpsHook" mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "FtrlOptimizer" mtype: "" } member { name: "GlobalStepWaiterHook" mtype: "" } member { name: "GradientDescentOptimizer" mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "LoggingTensorHook" mtype: "" } member { name: "LooperThread" mtype: "" } member { name: "MomentumOptimizer" mtype: "" } member { name: "MonitoredSession" mtype: "" } member { name: "NanLossDuringTrainingError" mtype: "" } member { name: "NanTensorHook" mtype: "" } member { name: "Optimizer" mtype: "" } member { name: "ProfilerHook" mtype: "" } member { name: "ProximalAdagradOptimizer" mtype: "" } member { name: "ProximalGradientDescentOptimizer" mtype: "" } member { name: "QueueRunner" mtype: "" } member { name: "RMSPropOptimizer" mtype: "" } member { name: "Saver" mtype: "" } member { name: "SaverDef" - mtype: "" + mtype: "" } member { name: "Scaffold" mtype: "" } member { name: "SecondOrStepTimer" mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "Server" mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "SessionCreator" mtype: "" } member { name: "SessionManager" mtype: "" } member { name: "SessionRunArgs" mtype: "" } member { name: "SessionRunContext" mtype: "" } member { name: "SessionRunHook" mtype: "" } member { name: "SessionRunValues" mtype: "" } member { name: "SingularMonitoredSession" mtype: "" } member { name: "StepCounterHook" mtype: "" } member { name: "StopAtStepHook" mtype: "" } member { name: "SummarySaverHook" mtype: "" } member { name: "Supervisor" mtype: "" } member { name: "SyncReplicasOptimizer" mtype: "" } member { name: "VocabInfo" mtype: "" } member { name: "WorkerSessionCreator" mtype: "" } member { name: "experimental" mtype: "" } member { name: "queue_runner" mtype: "" } member_method { name: "MonitoredTrainingSession" argspec: "args=[\'master\', \'is_chief\', \'checkpoint_dir\', \'scaffold\', \'hooks\', \'chief_only_hooks\', \'save_checkpoint_secs\', \'save_summaries_steps\', \'save_summaries_secs\', \'config\', \'stop_grace_period_secs\', \'log_step_count_steps\', \'max_wait_secs\', \'save_checkpoint_steps\', \'summary_dir\', \'save_graph_def\'], varargs=None, keywords=None, defaults=[\'\', \'True\', \'None\', \'None\', \'None\', \'None\', \'\', \'\', \'\', \'None\', \'120\', \'100\', \'7200\', \'\', \'None\', \'True\'], " } member_method { name: "NewCheckpointReader" argspec: "args=[\'filepattern\'], varargs=None, keywords=None, defaults=None" } member_method { name: "add_queue_runner" argspec: "args=[\'qr\', \'collection\'], varargs=None, keywords=None, defaults=[\'queue_runners\'], " } member_method { name: "assert_global_step" argspec: "args=[\'global_step_tensor\'], varargs=None, keywords=None, defaults=None" } member_method { name: "basic_train_loop" argspec: "args=[\'supervisor\', \'train_step_fn\', \'args\', \'kwargs\', \'master\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'\'], " } member_method { name: "batch" argspec: "args=[\'tensors\', \'batch_size\', \'num_threads\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "checkpoint_exists" argspec: "args=[\'checkpoint_prefix\'], varargs=None, keywords=None, defaults=None" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'alpha\', \'name\'], varargs=None, keywords=None, defaults=[\'0.0\', \'None\'], " } member_method { name: "cosine_decay_restarts" argspec: "args=[\'learning_rate\', \'global_step\', \'first_decay_steps\', \'t_mul\', \'m_mul\', \'alpha\', \'name\'], varargs=None, keywords=None, defaults=[\'2.0\', \'1.0\', \'0.0\', \'None\'], " } member_method { name: "create_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "do_quantize_training_on_graphdef" argspec: "args=[\'input_graph\', \'num_bits\'], varargs=None, keywords=None, defaults=None" } member_method { name: "exponential_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "export_meta_graph" argspec: "args=[\'filename\', \'meta_info_def\', \'graph_def\', \'saver_def\', \'collection_list\', \'as_text\', \'graph\', \'export_scope\', \'clear_devices\', \'clear_extraneous_savers\', \'strip_default_attrs\', \'save_debug_info\'], varargs=None, keywords=kwargs, defaults=[\'None\', \'None\', \'None\', \'None\', \'None\', \'False\', \'None\', \'None\', \'False\', \'False\', \'False\', \'False\'], " } member_method { name: "generate_checkpoint_state_proto" argspec: "args=[\'save_dir\', \'model_checkpoint_path\', \'all_model_checkpoint_paths\', \'all_model_checkpoint_timestamps\', \'last_preserved_timestamp\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_mtimes" argspec: "args=[\'checkpoint_prefixes\'], varargs=None, keywords=None, defaults=None" } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "get_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "get_or_create_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "global_step" argspec: "args=[\'sess\', \'global_step_tensor\'], varargs=None, keywords=None, defaults=None" } member_method { name: "import_meta_graph" argspec: "args=[\'meta_graph_or_file\', \'clear_devices\', \'import_scope\'], varargs=None, keywords=kwargs, defaults=[\'False\', \'None\'], " } member_method { name: "init_from_checkpoint" argspec: "args=[\'ckpt_dir_or_file\', \'assignment_map\'], varargs=None, keywords=None, defaults=None" } member_method { name: "input_producer" argspec: "args=[\'input_tensor\', \'element_shape\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'summary_name\', \'name\', \'cancel_op\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\', \'None\', \'32\', \'None\', \'None\', \'None\', \'None\'], " } member_method { name: "inverse_time_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "limit_epochs" argspec: "args=[\'tensor\', \'num_epochs\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "linear_cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'num_periods\', \'alpha\', \'beta\', \'name\'], varargs=None, keywords=None, defaults=[\'0.5\', \'0.0\', \'0.001\', \'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } member_method { name: "match_filenames_once" argspec: "args=[\'pattern\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "maybe_batch" argspec: "args=[\'tensors\', \'keep_input\', \'batch_size\', \'num_threads\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_batch_join" argspec: "args=[\'tensors_list\', \'keep_input\', \'batch_size\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_shuffle_batch" argspec: "args=[\'tensors\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'keep_input\', \'num_threads\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_shuffle_batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'keep_input\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "natural_exp_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "noisy_linear_cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'initial_variance\', \'variance_decay\', \'num_periods\', \'alpha\', \'beta\', \'name\'], varargs=None, keywords=None, defaults=[\'1.0\', \'0.55\', \'0.5\', \'0.0\', \'0.001\', \'None\'], " } member_method { name: "piecewise_constant" argspec: "args=[\'x\', \'boundaries\', \'values\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "piecewise_constant_decay" argspec: "args=[\'x\', \'boundaries\', \'values\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "polynomial_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'end_learning_rate\', \'power\', \'cycle\', \'name\'], varargs=None, keywords=None, defaults=[\'0.0001\', \'1.0\', \'False\', \'None\'], " } member_method { name: "range_input_producer" argspec: "args=[\'limit\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\'], " } member_method { name: "remove_checkpoint" argspec: "args=[\'checkpoint_prefix\', \'checkpoint_format_version\', \'meta_graph_suffix\'], varargs=None, keywords=None, defaults=[\'2\', \'meta\'], " } member_method { name: "replica_device_setter" argspec: "args=[\'ps_tasks\', \'ps_device\', \'worker_device\', \'merge_devices\', \'cluster\', \'ps_ops\', \'ps_strategy\'], varargs=None, keywords=None, defaults=[\'0\', \'/job:ps\', \'/job:worker\', \'True\', \'None\', \'None\', \'None\'], " } member_method { name: "sdca_fprint" argspec: "args=[\'input\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "sdca_optimizer" argspec: "args=[\'sparse_example_indices\', \'sparse_feature_indices\', \'sparse_feature_values\', \'dense_features\', \'example_weights\', \'example_labels\', \'sparse_indices\', \'sparse_weights\', \'dense_weights\', \'example_state_data\', \'loss_type\', \'l1\', \'l2\', \'num_loss_partitions\', \'num_inner_iterations\', \'adaptative\', \'name\'], varargs=None, keywords=None, defaults=[\'True\', \'None\'], " } member_method { name: "sdca_shrink_l1" argspec: "args=[\'weights\', \'l1\', \'l2\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "shuffle_batch" argspec: "args=[\'tensors\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'num_threads\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "shuffle_batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "slice_input_producer" argspec: "args=[\'tensor_list\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\'], " } member_method { name: "start_queue_runners" argspec: "args=[\'sess\', \'coord\', \'daemon\', \'start\', \'collection\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\', \'True\', \'queue_runners\'], " } member_method { name: "string_input_producer" argspec: "args=[\'string_tensor\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\', \'cancel_op\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\', \'None\'], " } member_method { name: "summary_iterator" argspec: "args=[\'path\'], varargs=None, keywords=None, defaults=None" } member_method { name: "update_checkpoint_state" argspec: "args=[\'save_dir\', \'model_checkpoint_path\', \'all_model_checkpoint_paths\', \'latest_filename\', \'all_model_checkpoint_timestamps\', \'last_preserved_timestamp\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'None\'], " } member_method { name: "warm_start" argspec: "args=[\'ckpt_to_initialize_from\', \'vars_to_warm_start\', \'var_name_to_vocab_info\', \'var_name_to_prev_var_name\'], varargs=None, keywords=None, defaults=[\'.*\', \'None\', \'None\'], " } member_method { name: "write_graph" argspec: "args=[\'graph_or_graph_def\', \'logdir\', \'name\', \'as_text\'], varargs=None, keywords=None, defaults=[\'True\'], " } } E0310 20:36:31.542584 281473543271296 api_compatibility_test.py:393] path: "tensorflow.train" tf_module { member { name: "AdadeltaOptimizer" mtype: "" } member { name: "AdagradDAOptimizer" mtype: "" } member { name: "AdagradOptimizer" mtype: "" } member { name: "AdamOptimizer" mtype: "" } member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointSaverHook" mtype: "" } member { name: "CheckpointSaverListener" mtype: "" } member { name: "ChiefSessionCreator" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FeedFnHook" mtype: "" } member { name: "FinalOpsHook" mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "FtrlOptimizer" mtype: "" } member { name: "GlobalStepWaiterHook" mtype: "" } member { name: "GradientDescentOptimizer" mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "LoggingTensorHook" mtype: "" } member { name: "LooperThread" mtype: "" } member { name: "MomentumOptimizer" mtype: "" } member { name: "MonitoredSession" mtype: "" } member { name: "NanLossDuringTrainingError" mtype: "" } member { name: "NanTensorHook" mtype: "" } member { name: "Optimizer" mtype: "" } member { name: "ProfilerHook" mtype: "" } member { name: "ProximalAdagradOptimizer" mtype: "" } member { name: "ProximalGradientDescentOptimizer" mtype: "" } member { name: "QueueRunner" mtype: "" } member { name: "RMSPropOptimizer" mtype: "" } member { name: "Saver" mtype: "" } member { name: "SaverDef" - mtype: "" + mtype: "" } member { name: "Scaffold" mtype: "" } member { name: "SecondOrStepTimer" mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "Server" mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "SessionCreator" mtype: "" } member { name: "SessionManager" mtype: "" } member { name: "SessionRunArgs" mtype: "" } member { name: "SessionRunContext" mtype: "" } member { name: "SessionRunHook" mtype: "" } member { name: "SessionRunValues" mtype: "" } member { name: "SingularMonitoredSession" mtype: "" } member { name: "StepCounterHook" mtype: "" } member { name: "StopAtStepHook" mtype: "" } member { name: "SummarySaverHook" mtype: "" } member { name: "Supervisor" mtype: "" } member { name: "SyncReplicasOptimizer" mtype: "" } member { name: "VocabInfo" mtype: "" } member { name: "WorkerSessionCreator" mtype: "" } member { name: "experimental" mtype: "" } member { name: "queue_runner" mtype: "" } member_method { name: "MonitoredTrainingSession" argspec: "args=[\'master\', \'is_chief\', \'checkpoint_dir\', \'scaffold\', \'hooks\', \'chief_only_hooks\', \'save_checkpoint_secs\', \'save_summaries_steps\', \'save_summaries_secs\', \'config\', \'stop_grace_period_secs\', \'log_step_count_steps\', \'max_wait_secs\', \'save_checkpoint_steps\', \'summary_dir\', \'save_graph_def\'], varargs=None, keywords=None, defaults=[\'\', \'True\', \'None\', \'None\', \'None\', \'None\', \'\', \'\', \'\', \'None\', \'120\', \'100\', \'7200\', \'\', \'None\', \'True\'], " } member_method { name: "NewCheckpointReader" argspec: "args=[\'filepattern\'], varargs=None, keywords=None, defaults=None" } member_method { name: "add_queue_runner" argspec: "args=[\'qr\', \'collection\'], varargs=None, keywords=None, defaults=[\'queue_runners\'], " } member_method { name: "assert_global_step" argspec: "args=[\'global_step_tensor\'], varargs=None, keywords=None, defaults=None" } member_method { name: "basic_train_loop" argspec: "args=[\'supervisor\', \'train_step_fn\', \'args\', \'kwargs\', \'master\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'\'], " } member_method { name: "batch" argspec: "args=[\'tensors\', \'batch_size\', \'num_threads\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "checkpoint_exists" argspec: "args=[\'checkpoint_prefix\'], varargs=None, keywords=None, defaults=None" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'alpha\', \'name\'], varargs=None, keywords=None, defaults=[\'0.0\', \'None\'], " } member_method { name: "cosine_decay_restarts" argspec: "args=[\'learning_rate\', \'global_step\', \'first_decay_steps\', \'t_mul\', \'m_mul\', \'alpha\', \'name\'], varargs=None, keywords=None, defaults=[\'2.0\', \'1.0\', \'0.0\', \'None\'], " } member_method { name: "create_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "do_quantize_training_on_graphdef" argspec: "args=[\'input_graph\', \'num_bits\'], varargs=None, keywords=None, defaults=None" } member_method { name: "exponential_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "export_meta_graph" argspec: "args=[\'filename\', \'meta_info_def\', \'graph_def\', \'saver_def\', \'collection_list\', \'as_text\', \'graph\', \'export_scope\', \'clear_devices\', \'clear_extraneous_savers\', \'strip_default_attrs\', \'save_debug_info\'], varargs=None, keywords=kwargs, defaults=[\'None\', \'None\', \'None\', \'None\', \'None\', \'False\', \'None\', \'None\', \'False\', \'False\', \'False\', \'False\'], " } member_method { name: "generate_checkpoint_state_proto" argspec: "args=[\'save_dir\', \'model_checkpoint_path\', \'all_model_checkpoint_paths\', \'all_model_checkpoint_timestamps\', \'last_preserved_timestamp\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_mtimes" argspec: "args=[\'checkpoint_prefixes\'], varargs=None, keywords=None, defaults=None" } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "get_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "get_or_create_global_step" argspec: "args=[\'graph\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "global_step" argspec: "args=[\'sess\', \'global_step_tensor\'], varargs=None, keywords=None, defaults=None" } member_method { name: "import_meta_graph" argspec: "args=[\'meta_graph_or_file\', \'clear_devices\', \'import_scope\'], varargs=None, keywords=kwargs, defaults=[\'False\', \'None\'], " } member_method { name: "init_from_checkpoint" argspec: "args=[\'ckpt_dir_or_file\', \'assignment_map\'], varargs=None, keywords=None, defaults=None" } member_method { name: "input_producer" argspec: "args=[\'input_tensor\', \'element_shape\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'summary_name\', \'name\', \'cancel_op\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\', \'None\', \'32\', \'None\', \'None\', \'None\', \'None\'], " } member_method { name: "inverse_time_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "limit_epochs" argspec: "args=[\'tensor\', \'num_epochs\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "linear_cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'num_periods\', \'alpha\', \'beta\', \'name\'], varargs=None, keywords=None, defaults=[\'0.5\', \'0.0\', \'0.001\', \'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } member_method { name: "match_filenames_once" argspec: "args=[\'pattern\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "maybe_batch" argspec: "args=[\'tensors\', \'keep_input\', \'batch_size\', \'num_threads\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_batch_join" argspec: "args=[\'tensors_list\', \'keep_input\', \'batch_size\', \'capacity\', \'enqueue_many\', \'shapes\', \'dynamic_pad\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'32\', \'False\', \'None\', \'False\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_shuffle_batch" argspec: "args=[\'tensors\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'keep_input\', \'num_threads\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "maybe_shuffle_batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'keep_input\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "natural_exp_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'decay_rate\', \'staircase\', \'name\'], varargs=None, keywords=None, defaults=[\'False\', \'None\'], " } member_method { name: "noisy_linear_cosine_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'initial_variance\', \'variance_decay\', \'num_periods\', \'alpha\', \'beta\', \'name\'], varargs=None, keywords=None, defaults=[\'1.0\', \'0.55\', \'0.5\', \'0.0\', \'0.001\', \'None\'], " } member_method { name: "piecewise_constant" argspec: "args=[\'x\', \'boundaries\', \'values\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "piecewise_constant_decay" argspec: "args=[\'x\', \'boundaries\', \'values\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "polynomial_decay" argspec: "args=[\'learning_rate\', \'global_step\', \'decay_steps\', \'end_learning_rate\', \'power\', \'cycle\', \'name\'], varargs=None, keywords=None, defaults=[\'0.0001\', \'1.0\', \'False\', \'None\'], " } member_method { name: "range_input_producer" argspec: "args=[\'limit\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\'], " } member_method { name: "remove_checkpoint" argspec: "args=[\'checkpoint_prefix\', \'checkpoint_format_version\', \'meta_graph_suffix\'], varargs=None, keywords=None, defaults=[\'2\', \'meta\'], " } member_method { name: "replica_device_setter" argspec: "args=[\'ps_tasks\', \'ps_device\', \'worker_device\', \'merge_devices\', \'cluster\', \'ps_ops\', \'ps_strategy\'], varargs=None, keywords=None, defaults=[\'0\', \'/job:ps\', \'/job:worker\', \'True\', \'None\', \'None\', \'None\'], " } member_method { name: "sdca_fprint" argspec: "args=[\'input\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "sdca_optimizer" argspec: "args=[\'sparse_example_indices\', \'sparse_feature_indices\', \'sparse_feature_values\', \'dense_features\', \'example_weights\', \'example_labels\', \'sparse_indices\', \'sparse_weights\', \'dense_weights\', \'example_state_data\', \'loss_type\', \'l1\', \'l2\', \'num_loss_partitions\', \'num_inner_iterations\', \'adaptative\', \'name\'], varargs=None, keywords=None, defaults=[\'True\', \'None\'], " } member_method { name: "sdca_shrink_l1" argspec: "args=[\'weights\', \'l1\', \'l2\', \'name\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "shuffle_batch" argspec: "args=[\'tensors\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'num_threads\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'1\', \'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "shuffle_batch_join" argspec: "args=[\'tensors_list\', \'batch_size\', \'capacity\', \'min_after_dequeue\', \'seed\', \'enqueue_many\', \'shapes\', \'allow_smaller_final_batch\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'False\', \'None\', \'False\', \'None\', \'None\'], " } member_method { name: "slice_input_producer" argspec: "args=[\'tensor_list\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\'], " } member_method { name: "start_queue_runners" argspec: "args=[\'sess\', \'coord\', \'daemon\', \'start\', \'collection\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\', \'True\', \'queue_runners\'], " } member_method { name: "string_input_producer" argspec: "args=[\'string_tensor\', \'num_epochs\', \'shuffle\', \'seed\', \'capacity\', \'shared_name\', \'name\', \'cancel_op\'], varargs=None, keywords=None, defaults=[\'None\', \'True\', \'None\', \'32\', \'None\', \'None\', \'None\'], " } member_method { name: "summary_iterator" argspec: "args=[\'path\'], varargs=None, keywords=None, defaults=None" } member_method { name: "update_checkpoint_state" argspec: "args=[\'save_dir\', \'model_checkpoint_path\', \'all_model_checkpoint_paths\', \'latest_filename\', \'all_model_checkpoint_timestamps\', \'last_preserved_timestamp\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'None\'], " } member_method { name: "warm_start" argspec: "args=[\'ckpt_to_initialize_from\', \'vars_to_warm_start\', \'var_name_to_vocab_info\', \'var_name_to_prev_var_name\'], varargs=None, keywords=None, defaults=[\'.*\', \'None\', \'None\'], " } member_method { name: "write_graph" argspec: "args=[\'graph_or_graph_def\', \'logdir\', \'name\', \'as_text\'], varargs=None, keywords=None, defaults=[\'True\'], " } } ERROR:tensorflow: Change detected in python object: tensorflow.profiler. E0310 20:36:31.542775 281473543271296 api_compatibility_test.py:392] Change detected in python object: tensorflow.profiler. ERROR:tensorflow: path: "tensorflow.profiler" tf_module { member { name: "AdviceProto" - mtype: "" + mtype: "" } member { name: "GraphNodeProto" - mtype: "" + mtype: "" } member { name: "MultiGraphNodeProto" - mtype: "" + mtype: "" } member { name: "OpLogProto" - mtype: "" + mtype: "" } member { name: "ProfileOptionBuilder" mtype: "" } member { name: "Profiler" mtype: "" } member_method { name: "advise" argspec: "args=[\'graph\', \'run_meta\', \'options\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'0\'], " } member_method { name: "profile" argspec: "args=[\'graph\', \'run_meta\', \'op_log\', \'cmd\', \'options\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'scope\', \'0\'], " } member_method { name: "write_op_log" argspec: "args=[\'graph\', \'log_dir\', \'op_log\', \'run_meta\', \'add_trace\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\'], " } } E0310 20:36:31.542912 281473543271296 api_compatibility_test.py:393] path: "tensorflow.profiler" tf_module { member { name: "AdviceProto" - mtype: "" + mtype: "" } member { name: "GraphNodeProto" - mtype: "" + mtype: "" } member { name: "MultiGraphNodeProto" - mtype: "" + mtype: "" } member { name: "OpLogProto" - mtype: "" + mtype: "" } member { name: "ProfileOptionBuilder" mtype: "" } member { name: "Profiler" mtype: "" } member_method { name: "advise" argspec: "args=[\'graph\', \'run_meta\', \'options\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'0\'], " } member_method { name: "profile" argspec: "args=[\'graph\', \'run_meta\', \'op_log\', \'cmd\', \'options\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'scope\', \'0\'], " } member_method { name: "write_op_log" argspec: "args=[\'graph\', \'log_dir\', \'op_log\', \'run_meta\', \'add_trace\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'True\'], " } } ERROR:tensorflow: Change detected in python object: tensorflow. E0310 20:36:31.543058 281473543271296 api_compatibility_test.py:392] Change detected in python object: tensorflow. ERROR:tensorflow: None : --- +++ @@ -10,7 +10,7 @@ } member { name: "AttrValue" - mtype: "" + mtype: "" } member { name: "COMPILER_VERSION" @@ -34,7 +34,7 @@ } member { name: "ConfigProto" - mtype: "" + mtype: "" } member { name: "CriticalSection" @@ -54,7 +54,7 @@ } member { name: "Event" - mtype: "" + mtype: "" } member { name: "FIFOQueue" @@ -78,7 +78,7 @@ } member { name: "GPUOptions" - mtype: "" + mtype: "" } member { name: "GRAPH_DEF_VERSION" @@ -102,7 +102,7 @@ } member { name: "GraphDef" - mtype: "" + mtype: "" } member { name: "GraphKeys" @@ -110,11 +110,11 @@ } member { name: "GraphOptions" - mtype: "" + mtype: "" } member { name: "HistogramProto" - mtype: "" + mtype: "" } member { name: "IdentityReader" @@ -138,7 +138,7 @@ } member { name: "LogMessage" - mtype: "" + mtype: "" } member { name: "MONOLITHIC_BUILD" @@ -146,7 +146,7 @@ } member { name: "MetaGraphDef" - mtype: "" + mtype: "" } member { name: "Module" @@ -154,11 +154,11 @@ } member { name: "NameAttrList" - mtype: "" + mtype: "" } member { name: "NodeDef" - mtype: "" + mtype: "" } member { name: "OpError" @@ -170,7 +170,7 @@ } member { name: "OptimizerOptions" - mtype: "" + mtype: "" } member { name: "OptionalSpec" @@ -214,11 +214,11 @@ } member { name: "RunMetadata" - mtype: "" + mtype: "" } member { name: "RunOptions" - mtype: "" + mtype: "" } member { name: "Session" @@ -226,7 +226,7 @@ } member { name: "SessionLog" - mtype: "" + mtype: "" } member { name: "SparseConditionalAccumulator" @@ -250,11 +250,11 @@ } member { name: "Summary" - mtype: "" + mtype: "" } member { name: "SummaryMetadata" - mtype: "" + mtype: "" } member { name: "TFRecordReader" @@ -274,7 +274,7 @@ } member { name: "TensorInfo" - mtype: "" + mtype: "" } member { name: "TensorShape" E0310 20:36:31.543189 281473543271296 api_compatibility_test.py:393] None : --- +++ @@ -10,7 +10,7 @@ } member { name: "AttrValue" - mtype: "" + mtype: "" } member { name: "COMPILER_VERSION" @@ -34,7 +34,7 @@ } member { name: "ConfigProto" - mtype: "" + mtype: "" } member { name: "CriticalSection" @@ -54,7 +54,7 @@ } member { name: "Event" - mtype: "" + mtype: "" } member { name: "FIFOQueue" @@ -78,7 +78,7 @@ } member { name: "GPUOptions" - mtype: "" + mtype: "" } member { name: "GRAPH_DEF_VERSION" @@ -102,7 +102,7 @@ } member { name: "GraphDef" - mtype: "" + mtype: "" } member { name: "GraphKeys" @@ -110,11 +110,11 @@ } member { name: "GraphOptions" - mtype: "" + mtype: "" } member { name: "HistogramProto" - mtype: "" + mtype: "" } member { name: "IdentityReader" @@ -138,7 +138,7 @@ } member { name: "LogMessage" - mtype: "" + mtype: "" } member { name: "MONOLITHIC_BUILD" @@ -146,7 +146,7 @@ } member { name: "MetaGraphDef" - mtype: "" + mtype: "" } member { name: "Module" @@ -154,11 +154,11 @@ } member { name: "NameAttrList" - mtype: "" + mtype: "" } member { name: "NodeDef" - mtype: "" + mtype: "" } member { name: "OpError" @@ -170,7 +170,7 @@ } member { name: "OptimizerOptions" - mtype: "" + mtype: "" } member { name: "OptionalSpec" @@ -214,11 +214,11 @@ } member { name: "RunMetadata" - mtype: "" + mtype: "" } member { name: "RunOptions" - mtype: "" + mtype: "" } member { name: "Session" @@ -226,7 +226,7 @@ } member { name: "SessionLog" - mtype: "" + mtype: "" } member { name: "SparseConditionalAccumulator" @@ -250,11 +250,11 @@ } member { name: "Summary" - mtype: "" + mtype: "" } member { name: "SummaryMetadata" - mtype: "" + mtype: "" } member { name: "TFRecordReader" @@ -274,7 +274,7 @@ } member { name: "TensorInfo" - mtype: "" + mtype: "" } member { name: "TensorShape" ERROR:tensorflow: Change detected in python object: tensorflow.summary. E0310 20:36:31.543368 281473543271296 api_compatibility_test.py:392] Change detected in python object: tensorflow.summary. ERROR:tensorflow: path: "tensorflow.summary" tf_module { member { name: "Event" - mtype: "" + mtype: "" } member { name: "FileWriter" mtype: "" } member { name: "FileWriterCache" mtype: "" } member { name: "SessionLog" - mtype: "" + mtype: "" } member { name: "Summary" - mtype: "" + mtype: "" } member { name: "SummaryDescription" - mtype: "" + mtype: "" } member { name: "TaggedRunMetadata" - mtype: "" + mtype: "" } member_method { name: "all_v2_summary_ops" argspec: "args=[], varargs=None, keywords=None, defaults=None" } member_method { name: "audio" argspec: "args=[\'name\', \'tensor\', \'sample_rate\', \'max_outputs\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'3\', \'None\', \'None\'], " } member_method { name: "get_summary_description" argspec: "args=[\'node_def\'], varargs=None, keywords=None, defaults=None" } member_method { name: "histogram" argspec: "args=[\'name\', \'values\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "image" argspec: "args=[\'name\', \'tensor\', \'max_outputs\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'3\', \'None\', \'None\'], " } member_method { name: "initialize" argspec: "args=[\'graph\', \'session\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "merge" argspec: "args=[\'inputs\', \'collections\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "merge_all" argspec: "args=[\'key\', \'scope\', \'name\'], varargs=None, keywords=None, defaults=[\'summaries\', \'None\', \'None\'], " } member_method { name: "scalar" argspec: "args=[\'name\', \'tensor\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "tensor_summary" argspec: "args=[\'name\', \'tensor\', \'summary_description\', \'collections\', \'summary_metadata\', \'family\', \'display_name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'None\', \'None\'], " } member_method { name: "text" argspec: "args=[\'name\', \'tensor\', \'collections\'], varargs=None, keywords=None, defaults=[\'None\'], " } } E0310 20:36:31.543499 281473543271296 api_compatibility_test.py:393] path: "tensorflow.summary" tf_module { member { name: "Event" - mtype: "" + mtype: "" } member { name: "FileWriter" mtype: "" } member { name: "FileWriterCache" mtype: "" } member { name: "SessionLog" - mtype: "" + mtype: "" } member { name: "Summary" - mtype: "" + mtype: "" } member { name: "SummaryDescription" - mtype: "" + mtype: "" } member { name: "TaggedRunMetadata" - mtype: "" + mtype: "" } member_method { name: "all_v2_summary_ops" argspec: "args=[], varargs=None, keywords=None, defaults=None" } member_method { name: "audio" argspec: "args=[\'name\', \'tensor\', \'sample_rate\', \'max_outputs\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'3\', \'None\', \'None\'], " } member_method { name: "get_summary_description" argspec: "args=[\'node_def\'], varargs=None, keywords=None, defaults=None" } member_method { name: "histogram" argspec: "args=[\'name\', \'values\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "image" argspec: "args=[\'name\', \'tensor\', \'max_outputs\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'3\', \'None\', \'None\'], " } member_method { name: "initialize" argspec: "args=[\'graph\', \'session\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "merge" argspec: "args=[\'inputs\', \'collections\', \'name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "merge_all" argspec: "args=[\'key\', \'scope\', \'name\'], varargs=None, keywords=None, defaults=[\'summaries\', \'None\', \'None\'], " } member_method { name: "scalar" argspec: "args=[\'name\', \'tensor\', \'collections\', \'family\'], varargs=None, keywords=None, defaults=[\'None\', \'None\'], " } member_method { name: "tensor_summary" argspec: "args=[\'name\', \'tensor\', \'summary_description\', \'collections\', \'summary_metadata\', \'family\', \'display_name\'], varargs=None, keywords=None, defaults=[\'None\', \'None\', \'None\', \'None\', \'None\'], " } member_method { name: "text" argspec: "args=[\'name\', \'tensor\', \'collections\'], varargs=None, keywords=None, defaults=[\'None\'], " } } INFO:tensorflow:time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibilityV1): 2.61s I0310 20:36:31.543720 281473543271296 test_util.py:2467] time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibilityV1): 2.61s [ FAILED ] ApiCompatibilityTest.testAPIBackwardsCompatibilityV1 [ RUN ] ApiCompatibilityTest.testAPIBackwardsCompatibilityV2 ERROR:tensorflow:TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. E0310 20:36:33.531735 281473543271296 api_compatibility_test.py:370] TensorFlow API backwards compatibility test This test ensures all changes to the public API of TensorFlow are intended. If this test fails, it means a change has been made to the public API. Backwards incompatible changes are not allowed. You can run the test as follows to update test goldens and package them with your change. $ bazel run tensorflow/tools/api/tests:api_compatibility_test \ # -- --update_goldens True You will need an API approval to make changes to the public TensorFlow API. This includes additions to the API. ERROR:tensorflow:1 differences found between API and golden. E0310 20:36:33.532064 281473543271296 api_compatibility_test.py:371] 1 differences found between API and golden. ERROR:tensorflow: Change detected in python object: tensorflow.train. E0310 20:36:33.532218 281473543271296 api_compatibility_test.py:392] Change detected in python object: tensorflow.train. ERROR:tensorflow: path: "tensorflow.train" tf_module { member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointView" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "TrackableView" mtype: "" } member { name: "experimental" mtype: "" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } } E0310 20:36:33.532355 281473543271296 api_compatibility_test.py:393] path: "tensorflow.train" tf_module { member { name: "BytesList" - mtype: "" + mtype: "" } member { name: "Checkpoint" mtype: "" } member { name: "CheckpointManager" mtype: "" } member { name: "CheckpointOptions" mtype: "" } member { name: "CheckpointView" mtype: "" } member { name: "ClusterDef" - mtype: "" + mtype: "" } member { name: "ClusterSpec" mtype: "" } member { name: "Coordinator" mtype: "" } member { name: "Example" - mtype: "" + mtype: "" } member { name: "ExponentialMovingAverage" mtype: "" } member { name: "Feature" - mtype: "" + mtype: "" } member { name: "FeatureList" - mtype: "" + mtype: "" } member { name: "FeatureLists" - mtype: "" + mtype: "" } member { name: "Features" - mtype: "" + mtype: "" } member { name: "FloatList" - mtype: "" + mtype: "" } member { name: "Int64List" - mtype: "" + mtype: "" } member { name: "JobDef" - mtype: "" + mtype: "" } member { name: "SequenceExample" - mtype: "" + mtype: "" } member { name: "ServerDef" - mtype: "" + mtype: "" } member { name: "TrackableView" mtype: "" } member { name: "experimental" mtype: "" } member_method { name: "checkpoints_iterator" argspec: "args=[\'checkpoint_dir\', \'min_interval_secs\', \'timeout\', \'timeout_fn\'], varargs=None, keywords=None, defaults=[\'0\', \'None\', \'None\'], " } member_method { name: "get_checkpoint_state" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "latest_checkpoint" argspec: "args=[\'checkpoint_dir\', \'latest_filename\'], varargs=None, keywords=None, defaults=[\'None\'], " } member_method { name: "list_variables" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_checkpoint" argspec: "args=[\'ckpt_dir_or_file\'], varargs=None, keywords=None, defaults=None" } member_method { name: "load_variable" argspec: "args=[\'ckpt_dir_or_file\', \'name\'], varargs=None, keywords=None, defaults=None" } } INFO:tensorflow:time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibilityV2): 1.99s I0310 20:36:33.532586 281473543271296 test_util.py:2467] time(__main__.ApiCompatibilityTest.testAPIBackwardsCompatibilityV2): 1.99s [ FAILED ] ApiCompatibilityTest.testAPIBackwardsCompatibilityV2 [ RUN ] ApiCompatibilityTest.testNoSubclassOfMessage INFO:tensorflow:time(__main__.ApiCompatibilityTest.testNoSubclassOfMessage): 0.62s I0310 20:36:34.157747 281473543271296 test_util.py:2467] time(__main__.ApiCompatibilityTest.testNoSubclassOfMessage): 0.62s [ OK ] ApiCompatibilityTest.testNoSubclassOfMessage [ RUN ] ApiCompatibilityTest.testNoSubclassOfMessageV1 INFO:tensorflow:time(__main__.ApiCompatibilityTest.testNoSubclassOfMessageV1): 0.28s I0310 20:36:34.433996 281473543271296 test_util.py:2467] time(__main__.ApiCompatibilityTest.testNoSubclassOfMessageV1): 0.28s [ OK ] ApiCompatibilityTest.testNoSubclassOfMessageV1 [ RUN ] ApiCompatibilityTest.testNoSubclassOfMessageV2 INFO:tensorflow:time(__main__.ApiCompatibilityTest.testNoSubclassOfMessageV2): 0.29s I0310 20:36:34.726405 281473543271296 test_util.py:2467] time(__main__.ApiCompatibilityTest.testNoSubclassOfMessageV2): 0.29s [ OK ] ApiCompatibilityTest.testNoSubclassOfMessageV2 [ RUN ] ApiCompatibilityTest.test_session [ SKIPPED ] ApiCompatibilityTest.test_session ====================================================================== FAIL: testAPIBackwardsCompatibility (__main__.ApiCompatibilityTest) ApiCompatibilityTest.testAPIBackwardsCompatibility ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 511, in testAPIBackwardsCompatibility self._checkBackwardsCompatibility( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 488, in _checkBackwardsCompatibility self._AssertProtoDictEquals( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 395, in _AssertProtoDictEquals self.fail('%d differences found between API and golden.' % diff_count) AssertionError: 1 differences found between API and golden. ====================================================================== FAIL: testAPIBackwardsCompatibilityV1 (__main__.ApiCompatibilityTest) ApiCompatibilityTest.testAPIBackwardsCompatibilityV1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 528, in testAPIBackwardsCompatibilityV1 self._checkBackwardsCompatibility( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 488, in _checkBackwardsCompatibility self._AssertProtoDictEquals( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 395, in _AssertProtoDictEquals self.fail('%d differences found between API and golden.' % diff_count) AssertionError: 4 differences found between API and golden. ====================================================================== FAIL: testAPIBackwardsCompatibilityV2 (__main__.ApiCompatibilityTest) ApiCompatibilityTest.testAPIBackwardsCompatibilityV2 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 550, in testAPIBackwardsCompatibilityV2 self._checkBackwardsCompatibility( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 488, in _checkBackwardsCompatibility self._AssertProtoDictEquals( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/tools/api/tests/api_compatibility_test.runfiles/org_tensorflow/tensorflow/tools/api/tests/api_compatibility_test.py", line 395, in _AssertProtoDictEquals self.fail('%d differences found between API and golden.' % diff_count) AssertionError: 1 differences found between API and golden. ---------------------------------------------------------------------- Ran 7 tests in 8.022s FAILED (failures=3, skipped=1) ================================================================================ //tensorflow/c:c_api_experimental_test PASSED in 27.4s //tensorflow/c:c_api_function_test PASSED in 29.6s //tensorflow/c:c_api_test_cpu PASSED in 30.2s //tensorflow/c:c_test PASSED in 28.0s //tensorflow/c:env_test_cpu PASSED in 26.8s //tensorflow/c:kernels_test_cpu PASSED in 34.6s //tensorflow/c:ops_test PASSED in 23.2s //tensorflow/c:while_loop_test PASSED in 27.1s //tensorflow/c/eager:c_api_cluster_test_cpu PASSED in 30.3s //tensorflow/c/eager:c_api_remote_function_test_cpu PASSED in 28.7s //tensorflow/c/eager:c_api_remote_test_cpu PASSED in 30.0s //tensorflow/c/eager:c_api_test_cpu PASSED in 24.8s //tensorflow/c/eager:custom_device_test PASSED in 27.0s //tensorflow/c/eager/parallel_device:parallel_device_lib_test PASSED in 26.9s //tensorflow/c/eager/parallel_device:parallel_device_remote_test PASSED in 28.5s //tensorflow/c/eager/parallel_device:parallel_device_test PASSED in 28.4s //tensorflow/c/experimental/filesystem/plugins/gcs:expiring_lru_cache_test PASSED in 0.1s //tensorflow/c/experimental/filesystem/plugins/gcs:ram_file_block_cache_test PASSED in 12.9s //tensorflow/c/experimental/grappler:grappler_test PASSED in 23.0s //tensorflow/c/experimental/ops/gen/common:case_format_test PASSED in 0.5s //tensorflow/c/experimental/ops/gen/cpp:cpp_generator_test PASSED in 0.6s //tensorflow/c/experimental/ops/gen/cpp/renderers:renderer_test PASSED in 0.5s //tensorflow/c/experimental/saved_model/core:constant_loading_test PASSED in 13.5s //tensorflow/c/experimental/saved_model/core:object_graph_traversal_test PASSED in 11.7s //tensorflow/c/experimental/saved_model/core:saved_variable_loading_test PASSED in 18.3s //tensorflow/c/experimental/saved_model/core:signature_flattening_test PASSED in 11.1s //tensorflow/c/experimental/saved_model/core:tf_concrete_function_loading_test PASSED in 14.4s //tensorflow/c/experimental/saved_model/core/ops:restore_ops_test PASSED in 13.0s //tensorflow/c/experimental/saved_model/core/ops:variable_ops_test PASSED in 12.8s //tensorflow/c/experimental/saved_model/internal:saved_model_api_test PASSED in 29.7s //tensorflow/c/experimental/stream_executor:stream_executor_test PASSED in 1.1s //tensorflow/c/kernels:bitcast_op_test PASSED in 0.9s //tensorflow/c/kernels:summary_op_benchmark_test PASSED in 0.6s //tensorflow/c/kernels:summary_op_test PASSED in 1.1s //tensorflow/c/kernels:tensor_shape_utils_test PASSED in 0.1s //tensorflow/cc:cc_op_gen_test PASSED in 0.6s //tensorflow/cc:client_client_session_test PASSED in 2.3s //tensorflow/cc:coordinator_test PASSED in 4.0s //tensorflow/cc:framework_cc_ops_test PASSED in 2.0s //tensorflow/cc:framework_gradient_checker_test PASSED in 2.7s //tensorflow/cc:framework_gradients_test PASSED in 5.0s //tensorflow/cc:framework_scope_test PASSED in 0.7s //tensorflow/cc:framework_while_gradients_test PASSED in 3.0s //tensorflow/cc:gradients_array_grad_test PASSED in 5.8s //tensorflow/cc:gradients_data_flow_grad_test PASSED in 2.0s //tensorflow/cc:gradients_functional_grad_test PASSED in 5.5s //tensorflow/cc:gradients_image_grad_test PASSED in 6.7s //tensorflow/cc:gradients_linalg_grad_test PASSED in 2.7s //tensorflow/cc:gradients_manip_grad_test PASSED in 1.8s //tensorflow/cc:gradients_math_grad_test PASSED in 5.9s //tensorflow/cc:gradients_nn_grad_test PASSED in 3.1s //tensorflow/cc:gradients_resource_variable_grad_test PASSED in 2.3s //tensorflow/cc:ops_const_op_test PASSED in 0.9s //tensorflow/cc:ops_while_loop_test PASSED in 2.3s //tensorflow/cc:queue_runner_test PASSED in 12.8s //tensorflow/cc/experimental/base/tests:tensor_test PASSED in 0.2s //tensorflow/cc/experimental/base/tests:tensorhandle_test PASSED in 28.2s //tensorflow/cc/experimental/libexport:load_test PASSED in 0.2s //tensorflow/cc/experimental/libexport:save_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_module_test PASSED in 28.1s //tensorflow/cc/experimental/libtf:libtf_object_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_perf_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_runtime_test PASSED in 37.3s //tensorflow/cc/experimental/libtf:libtf_transform_test PASSED in 28.5s //tensorflow/cc/experimental/libtf:libtf_value_test PASSED in 0.3s //tensorflow/cc/experimental/libtf:libtf_visit_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:iostream_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:none_test PASSED in 0.2s //tensorflow/cc/experimental/libtf/impl:scalars_test PASSED in 0.2s //tensorflow/cc/experimental/libtf/impl:string_test PASSED in 0.3s //tensorflow/cc/experimental/libtf/impl:tensor_spec_test PASSED in 0.1s //tensorflow/cc/saved_model:bundle_v2_test PASSED in 0.1s //tensorflow/cc/saved_model:fingerprinting_test PASSED in 2.3s //tensorflow/cc/saved_model:metrics_test PASSED in 0.2s //tensorflow/cc/saved_model:reader_test PASSED in 0.2s //tensorflow/cc/saved_model:saved_model_bundle_lite_test PASSED in 6.1s //tensorflow/cc/saved_model:saved_model_bundle_test PASSED in 5.8s //tensorflow/cc/saved_model:util_test PASSED in 0.2s //tensorflow/cc/saved_model/experimental/tests:saved_model_api_test PASSED in 27.7s //tensorflow/cc/tools:freeze_saved_model_test PASSED in 1.9s //tensorflow/compiler/aot:codegen_test PASSED in 35.1s //tensorflow/compiler/jit:compilability_check_util_test PASSED in 18.2s //tensorflow/compiler/jit:deadness_analysis_test PASSED in 10.3s //tensorflow/compiler/jit:device_compilation_cache_test PASSED in 5.3s //tensorflow/compiler/jit:device_compilation_cluster_signature_test PASSED in 5.6s //tensorflow/compiler/jit:device_compilation_profiler_test PASSED in 17.9s //tensorflow/compiler/jit:device_compiler_client_test PASSED in 5.2s //tensorflow/compiler/jit:device_compiler_disable_test PASSED in 17.2s //tensorflow/compiler/jit:device_executable_persistor_test PASSED in 21.6s //tensorflow/compiler/jit:device_util_test PASSED in 6.0s //tensorflow/compiler/jit:encapsulate_util_test PASSED in 0.6s //tensorflow/compiler/jit:node_matchers_test PASSED in 0.6s //tensorflow/compiler/jit:resource_operation_safety_analysis_test PASSED in 9.1s //tensorflow/compiler/jit:shape_inference_test PASSED in 0.5s //tensorflow/compiler/jit:xla_activity_listener_test PASSED in 18.9s //tensorflow/compiler/jit:xla_cluster_util_test PASSED in 7.9s //tensorflow/compiler/jit:xla_compile_util_test PASSED in 5.1s //tensorflow/compiler/jit:xla_kernel_creator_test PASSED in 8.9s //tensorflow/compiler/jit/tests:auto_clustering_test PASSED in 22.2s //tensorflow/compiler/mlir:mlir_graph_optimization_pass_test PASSED in 12.5s //tensorflow/compiler/mlir:register_common_dialects_test PASSED in 13.8s //tensorflow/compiler/mlir/lite:lstm_utils_test PASSED in 0.8s //tensorflow/compiler/mlir/lite:perception_ops_utils_test PASSED in 0.5s //tensorflow/compiler/mlir/lite:size_utils_test PASSED in 0.2s //tensorflow/compiler/mlir/lite:tftext_utils_test PASSED in 0.9s //tensorflow/compiler/mlir/lite/experimental/tac:execution_metadata_exporter_test PASSED in 5.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:compute-cost.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-gpu.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-nnapi.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/experimental/tac/tests:fold-constants-to-subgraph.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-alternative-subgraph.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-op-cost.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests:pick-subgraphs.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/experimental/tac/tests:raise-target-subgraphs.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/experimental/tac/tests:target-annotation.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:device-transform-nnapi.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:simple-graph.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/metrics:error_collector_inst_test PASSED in 1.1s //tensorflow/compiler/mlir/lite/quantization:numerical_utils_test PASSED in 0.2s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_model_test PASSED in 9.5s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_weights_test PASSED in 9.1s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_default.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_legacy.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant_4bit.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/quantization/tests:import_quant_stats.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/sparsity:sparsify_model_test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:fold_broadcast.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:fuse_mhlo_convolution.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-inplaceupdate.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-skip-quantization-ops.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tf-fb-tf.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-add.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-broadcast_in_dim.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-clamp.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-compare.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-concat.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-constant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-conv.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-dot.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-gather.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-max.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-mul.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-pad.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-reshape.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-rsqrt.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-scatter.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-sub.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-add.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-broadcast.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-clamp.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-concat.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-constant.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-conv.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-max.mlir.test PASSED in 3.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-mul.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-pad.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-reshape.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-rsqrt.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-sub.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-allow-tf.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-smuggle-resize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:optimize.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-clamp.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-concat.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-conv.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-division.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-logistic.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-multiply.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-reduce-window.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-resize-bilinear.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-subtract.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-tf-quantize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:unfuse_mhlo_batch_norm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:analyze-variables.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:canonicalize.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:decompose-hybrid-quantization.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:default_quant_params.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:dilated-conv.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:fuse-tftext.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:get-arithmetic-count.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:guarantee_func_has_one_use.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests:inlining.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:insert_call_once_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:legalize-tf-assert.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:legalize-tf-hashtables.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:legalize-tf-no-runtime-verification.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:legalize-tf-variables.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:legalize-tf-while.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:legalize-tf.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests:legalize_jax_random.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:lift_tflite_flex_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-default-to-single-batch.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-enable-dynamic-update-slice.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:modify_io_nodes.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:optimize-after-quantization.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:optimize.mlir.test PASSED in 3.0s //tensorflow/compiler/mlir/lite/tests:optimize_functional_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:optimize_no_verify.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:optimize_op_order.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:partitioned-topological-sort.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:pin-ops-with-side-effects.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:post-quantize-dynamic-range.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:post-quantize.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:prepare-composite-functions-tf.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-dynamic-range.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training-16bits.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-signed.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:prepare-quantize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant-4bit.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:prepare-tf-with-allowing-bf16-and-f16-type-legalization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:prepare-tf.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:quantize-dynamic-range.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests:quantize-numeric-verify.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:quantize-variables.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:quantize.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:raise-custom-ops.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:reduce_while_operands.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:shape-inference.mlir.test PASSED in 11.9s //tensorflow/compiler/mlir/lite/tests:split-merged-operands.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:tfl_while_op_licm.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests:tfl_while_outline.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:trim-functions-tf.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:unfold-large-splat-constant.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.line.part.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.stack.part.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:add.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:back2back_fake_quant.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/end2end:control_flow_v1.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d_nchw.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/end2end:custom_opdef.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel_4bit.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity_4bit.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/end2end:graph-input-node.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:graph_with_placeholder_with_default.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:if_op.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:quant_stats.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul_disabled.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:basic_lstm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:bucketize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:control_edges.mlir.test PASSED in 0.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:dynamic_shape.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:external_constant.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:if_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:import_json.json.test PASSED in 8.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_arrays.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_output_names_attr.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:legacy_reshape.json.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.json.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:many_attribute_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:math.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:matmul.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:multi_output_op.json.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional_input.json.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:output_arrays.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning_function_input_as_output.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quant_stats.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quantization.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:reshape.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature_with_multiple_entry_points.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:simple.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:tf_variant_type.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_function_output.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_tensor.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:while_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2exec:tfl_while_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:basic_lstm.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:bucketize.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_op_with_tflite_op.mlir.test PASSED in 0.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d_v2.mlir.test PASSED in 0.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_builtin.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_custom.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex_enable_builtin.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:dynamic_shape_constant.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fake_quant.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_exclusively.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_complex128.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_f64.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_tflite_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected_v2.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:hashtable_resource.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:if_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:logical.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:low_bit_packing.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_asym_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_quantized.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:math.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:metadata.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v2.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v3.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:nn.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:numeric_verify.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:optional.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:quantization.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:reshape.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_output_override.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_multiple_entry_points.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_no_inputs.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_connected_control_nodes.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_unconnected_control_nodes.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf.mlir.test PASSED in 0.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf_v2.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tf_entry_function.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tfl_while_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:transpose_conv_optional.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:type_attr.mlir.test PASSED in 0.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_lstm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_rnn.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unranked_tensor.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unsorted_segment_prod.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_func.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:while_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibrator_singleton_test PASSED in 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:custom_aggregator_op_test PASSED in 19.0s //tensorflow/compiler/mlir/quantization/tensorflow/cc:const_op_size_test PASSED in 0.4s //tensorflow/compiler/mlir/quantization/tensorflow/cc:convert_asset_args_test PASSED in 4.7s //tensorflow/compiler/mlir/quantization/tensorflow/cc:save_variables_test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/cc:status_macro_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/tensorflow/debugging:mlir_dump_test PASSED in 0.2s //tensorflow/compiler/mlir/quantization/tensorflow/python:concurrency_test PASSED in 35.5s //tensorflow/compiler/mlir/quantization/tensorflow/python:pywrap_quantize_model_test PASSED in 15.7s //tensorflow/compiler/mlir/quantization/tensorflow/python:representative_dataset_test PASSED in 7.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:cast_bf16_ops_to_f32.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_custom_aggregation_op_to_quant_stats.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_fake_quant_to_qdq.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tf_quant_ops_to_mhlo.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tpu_model_to_cpu.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:duplicate_shape_determining_constants.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_flow.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_xla.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_custom_aggregation_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_main_function.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_drq.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_weight_only.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_restore_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_save_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:issue_ids_of_custom_aggregation_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq_min_elements.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_xla.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:mark_functions_noinline.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_initializer_function_ops_to_main.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_save_function_ops_to_main.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:optimize.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_lifting.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq_per_channel.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq_per_channel.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_drq.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_weight_only.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_xla.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_drq.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_xla.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:remove_var_init_by_const.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops_large_constants.mlir.test PASSED in 15.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:unfreeze_constants.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/utils:tf_to_xla_attribute_utils_test PASSED in 30.2s //tensorflow/compiler/mlir/tensorflow:bridge_logger_test PASSED in 4.2s //tensorflow/compiler/mlir/tensorflow:cluster_util_test PASSED in 0.4s //tensorflow/compiler/mlir/tensorflow:convert_tensor_test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow:convert_type_test PASSED in 0.2s //tensorflow/compiler/mlir/tensorflow:device_util_test PASSED in 0.3s //tensorflow/compiler/mlir/tensorflow:dump_graph_test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow:dump_mlir_util_test PASSED in 9.1s //tensorflow/compiler/mlir/tensorflow:error_util_test PASSED in 0.2s //tensorflow/compiler/mlir/tensorflow:tf_saved_model_test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow:tpu_rewrite_device_util_test PASSED in 0.3s //tensorflow/compiler/mlir/tensorflow/tests:add_functions_for_exported_names.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:annotate-parameter-replication.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:batchmatmul_to_einsum.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:breakup-islands.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:cannonicalize_ops_outside_compilation.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize_compile_and_replicate_attributes.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:check_control_dependencies.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:cluster_formation.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:cluster_ops_by_policy.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:cluster_outlining.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:cluster_tf_ops_pass.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:constant-fold.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:constant_op_device_assignment.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:convert-tf-control-flow-to-scf.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:convert_control_to_data_outputs.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:convert_launch_func_to_tf_call.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:convert_session_initializer_to_function.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:convert_to_legacy_compile_and_replicate_attributes.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:decompose_reduce_dataset.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:decompose_resource_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment_by_func_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:device_attribute_to_launch.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:device_canonicalize.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:device_copy.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:drop_while_shape_invariant.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:einsum.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:empty-main.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:end-to-end-tpu-reshard-variables.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:executor_canonicalize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_coarsening.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_materialize_const.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:extract_head_tail_outside_compilation.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:extract_outside_compilation.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:fold-broadcast.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:freeze_variables.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:func-attr-invalid.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:func-attr.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-cfg.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-regions.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if-fail.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:fused_kernel_matcher.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:gpu_fusion.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning_preserve_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:group_by_dialect.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:guarantee-all-funcs-one-use.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:hoist_loop_invariant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:hoist_replicate_invariant_resource_writes.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:host_launch_to_outside_compiled.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_invalid.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_saved_model.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:inlining.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:isolate-placer.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:launch_outlining.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute_legacy.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_60.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_70.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nchw.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nhwc.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_begin.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_end.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nchw.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nhwc.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:legalize_hlo.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_arg_control_dep.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_with_control_flow.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:localize_var_handles.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program_invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:lower_quantized.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:lower_tf.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:lower_variable_ops_to_ml_program.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:mark_input_output_aliases.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:mark_ops_for_outside_compilation.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:materialize_passthrough_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:merge_control_flow.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:mlprogram.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:name_anonymous_iterators.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:optimize-arg-operand-constraint.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:optimize.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:order_by_dialect.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:outside_compiled_to_host_launch.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands_legacy.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:prepare_tpu_computation_for_tf_export.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args_functions.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:promote_var_handles_to_args.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:readonly_references_to_resources.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:region-control-flow-to-functional.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_arguments.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_while_results.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:replica_id_to_device_ordinal.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:replicate_invariant_op_hoisting.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:replicate_tensor_list_init_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island_legacy.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:resource-alias-analysis-test.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:resource-device-inference.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:resource_analyzer.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:resource_inlining.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:resource_op_lifting.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:rewrite_tpu_embedding_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:roundtrip-tf-executor.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:shape_inference.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:side-effect-analysis-test.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:sink_constant.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:split_into_island_per_op.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:stack_ops_decomposition.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:strip_noinline.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:strip_saved_module_metadata.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:strip_tf_attributes.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tensor_array_ops_decomposition.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tensor_list_ops_decomposition.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:tf-executor-to-functional.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf-functional-to-executor.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf-ops.mlir.test PASSED in 2.8s //tensorflow/compiler/mlir/tensorflow/tests:tf-reduce-identity.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_map_and_batch.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_pmap_and_batch.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_index_selector.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops_invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_invalid.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_location_roundtrip.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_printer.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_side_effect.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_optimize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_deduplicate_bound_input_bindings.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_assets.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors_mutable_tensors.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init_fail.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables_invalid_session.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_mark_initialized_variables.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops_invalid.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors_interprocedural.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_remove_vars_in_session_initializer.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_side_effect.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_trait_folds.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu-cluster-cleanup-attributes.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tpu-dynamic-layout-pass.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tpu-merge-variables-with-execute.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu-multiple-while-body-func.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu-resource-read-for-write.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu-variable-runtime-reformatting.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_cluster_formation.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_composite_resource_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_device_propagation.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_host_computation_expansion.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tpu_identity_pruning.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_parallel_execute_sink_resource_write.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests:tpu_partitioned_op_conversion.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_reorder_replicate_and_partitioned_inputs.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_resource_partitioning.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_rewrite.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:tpu_sharding_identification.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_space_to_depth_pass.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_tail_with_tobool_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_update_embedding_enqueue_op_inputs.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_validate_inputs.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:unroll-batch-matmul.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:update_control_dependencies.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:warn_when_using_deprecated_dumps.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:while_licm.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:xla_cluster_formation.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:xla_inline_device_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:add.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding-invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding-hook.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph.pbtxt.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:mlir-module-serialized-str-attr.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:replicate-tensor-list-init-ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:result-sharding.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr-invalid.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference-after-legalization.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:stablehlo_add.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:executor_tpuv1_island_coarsening.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:while_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:executor_tpuv1_inline_tpu_island.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:while_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:case_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:executor_tpuv1_outline_tpu_island.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:while_op.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:add.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-as-fetch.pbtxt.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-control-dep.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type-with-subtype.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-multi-data-type-with-subtype.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-retval-attrs.pbtxt.test PASSED in 3.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:case_op.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:const-values.pbtxt.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:device-arg-retval-attr.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-input-shapes.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-value-attr.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-as-fetch.pbtxt.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-control-dep.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:force_shared_name_for_resource_ops.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:function-func-attr.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-if-ops.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-while-ops.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-control-ret.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-retval-of-arg.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-custom-operation.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-default-attr.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-device-retval.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-empty-tensor-content.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-func-attr.pbtxt.test PASSED in 3.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-call.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-diff-island.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-same-island.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-defs.pbtxt.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-input-shapes.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-name-bug.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-resource-args.pbtxt.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-gradient-def.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-input-func-arg-name-collision.pbtxt.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-library.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-malformed.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-scalar-input.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-uint8-return.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-undefined-output.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-version-info.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-while-loop.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:invalid-output-index.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:legacy-fed-input-without-inputs.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:merge_node_with_function.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:mlir_passthrough_op.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multi-output-feeds.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multiple-use-next-iteration.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:node-locations.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes-attr.pbtxt.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example_v2.pbtxt.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:partial-device-name.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:prune_unused_nodes.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:quint8-const.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:shape-attrs.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:stateful-attribute.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:string-attr.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:switch_n.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:target.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tensor-list.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tf-data-pipeline.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:unregistered_kernel.pbtxt.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/batch_use_same_function:saved_model.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:aliasing_arg_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:case.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:convert_tensor.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_shape_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_size_attr.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:device-arg-retval-attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:export_main_to_flib.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:fetch_feed_names.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_list_attr.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-control-ret.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-order.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args-handle-info.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-if-ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-while-ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:graph-as-function.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:infer_derived_attribute.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:invalid_input.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:legalized_name.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:missing-main.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:noop.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:optional_symbol_ref.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:output-shapes-attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example_v2.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:preserve-entry-func-names.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-type-attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-while-loop.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:shape_list_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple_tf_dialect_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:stringescape.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:switchn.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-gradient-attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-legacy-call.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_add.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_identity_n.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_tpu_embedding_ops.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_list_attr.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_name.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_output_name.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:while-loop.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/tf_to_hlo_pipeline:sccp-post-shape-inference.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/tpu_bridge_v1:end_to_end.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla:compile_mlir_util_test PASSED in 5.9s //tensorflow/compiler/mlir/tf2xla:verify_tfxla_legalization_test PASSED in 3.7s //tensorflow/compiler/mlir/tf2xla:xla_legalize_targets_test PASSED in 0.5s //tensorflow/compiler/mlir/tf2xla/tests:adjust-layout.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tf2xla/tests:convert-mhlo-quant-to-int.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_runtime_pipeline.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_sparsification.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-BatchMatMulV2.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-binary-elementwise.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-collective.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-communication.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-include-tf2xla-fallback.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-no-tf2xla-fallback.mlir.test PASSED in 5.3s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-prefer-tf2xla.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-types.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-with-tf2xla.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf.mlir.test PASSED in 10.2s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_cpu.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_gpu.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization-no-chlo.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfr:graph_decompose_test PASSED in 10.8s //tensorflow/compiler/mlir/tfr:node_expansion_test PASSED in 9.2s //tensorflow/compiler/mlir/tfr:op_reg_gen_test PASSED in 17.2s //tensorflow/compiler/mlir/tfr:tfr_decompose_ctx_test PASSED in 4.0s //tensorflow/compiler/mlir/tfr:tfr_gen_test PASSED in 14.6s //tensorflow/compiler/mlir/tfr/examples/customization:test_ops_test PASSED in 20.5s //tensorflow/compiler/mlir/tfr/examples/mnist:mnist_ops_test PASSED in 42.7s //tensorflow/compiler/mlir/tfr/examples/pad:pad_ops_test PASSED in 22.5s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_deallocation.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_reuse.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tools/kernel_gen/tests:bufferize.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:copy_cleanup.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:embed_tf_framework.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tools/kernel_gen/tests:invalid.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:isinf.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:ops.mlir.test PASSED in 3.3s //tensorflow/compiler/mlir/tools/kernel_gen/tests:parallel_loops_to_sequential.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:rewrite_tf_framework_assert.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tanh.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf-legalize-to-lmhlo.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_abi_knowledge.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_framework_legalize_to_llvm.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_kernel_gpu_launch_to_llvm.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_to_jit_invocations.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tosa/tests:convert-tfl-uint8.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:fuse-bias-tf.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:lower-complex-types.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tosa/tests:strip-quant-types.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:tf-tfl-to-tosa-pipeline.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:tf-to-tosa-pipeline.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-dequantize_softmax.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline-filtered.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline.mlir.test PASSED in 5.1s //tensorflow/compiler/tests:adadelta_test_cpu PASSED in 15.1s //tensorflow/compiler/tests:adagrad_da_test_cpu PASSED in 24.7s //tensorflow/compiler/tests:adagrad_test_cpu PASSED in 10.3s //tensorflow/compiler/tests:adam_test_cpu PASSED in 17.8s //tensorflow/compiler/tests:add_n_test_cpu PASSED in 12.0s //tensorflow/compiler/tests:argminmax_test_cpu PASSED in 12.4s //tensorflow/compiler/tests:argminmax_test_cpu_mlir_bridge_test PASSED in 15.8s //tensorflow/compiler/tests:bucketize_op_test_cpu PASSED in 7.8s //tensorflow/compiler/tests:bucketize_op_test_cpu_mlir_bridge_test PASSED in 7.5s //tensorflow/compiler/tests:case_test_cpu PASSED in 7.2s //tensorflow/compiler/tests:cast_ops_test_cpu PASSED in 7.2s //tensorflow/compiler/tests:cast_ops_test_cpu_mlir_bridge_test PASSED in 7.2s //tensorflow/compiler/tests:categorical_op_test_cpu PASSED in 20.4s //tensorflow/compiler/tests:categorical_op_test_cpu_mlir_bridge_test PASSED in 11.8s //tensorflow/compiler/tests:cholesky_op_test_cpu PASSED in 20.7s //tensorflow/compiler/tests:cholesky_op_test_cpu_mlir_bridge_test PASSED in 18.7s //tensorflow/compiler/tests:clustering_test_cpu PASSED in 7.0s //tensorflow/compiler/tests:clustering_test_cpu_mlir_bridge_test PASSED in 7.3s //tensorflow/compiler/tests:concat_ops_test_cpu PASSED in 7.8s //tensorflow/compiler/tests:concat_ops_test_cpu_mlir_bridge_test PASSED in 11.9s //tensorflow/compiler/tests:cond_test_cpu PASSED in 12.7s //tensorflow/compiler/tests:const_arg_test_cpu PASSED in 6.5s //tensorflow/compiler/tests:const_test_cpu PASSED in 10.8s //tensorflow/compiler/tests:data_format_ops_test_cpu PASSED in 10.1s //tensorflow/compiler/tests:data_format_ops_test_cpu_mlir_bridge_test PASSED in 11.9s //tensorflow/compiler/tests:dense_layer_test_cpu PASSED in 18.3s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu PASSED in 10.4s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu_mlir_bridge_test PASSED in 13.3s //tensorflow/compiler/tests:dynamic_stitch_test_cpu PASSED in 6.5s //tensorflow/compiler/tests:dynamic_stitch_test_cpu_mlir_bridge_test PASSED in 8.8s //tensorflow/compiler/tests:eager_test_cpu PASSED in 19.2s //tensorflow/compiler/tests:einsum_op_test_cpu PASSED in 7.2s //tensorflow/compiler/tests:einsum_op_test_cpu_mlir_bridge_test PASSED in 8.2s //tensorflow/compiler/tests:ensure_shape_op_test_cpu PASSED in 8.2s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu PASSED in 8.5s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu_mlir_bridge_test PASSED in 8.3s //tensorflow/compiler/tests:fake_quant_ops_test_cpu PASSED in 13.7s //tensorflow/compiler/tests:fake_quant_ops_test_cpu_mlir_bridge_test PASSED in 14.3s //tensorflow/compiler/tests:fifo_queue_test_cpu PASSED in 8.7s //tensorflow/compiler/tests:fifo_queue_test_cpu_mlir_bridge_test PASSED in 10.8s //tensorflow/compiler/tests:ftrl_ops_test_cpu PASSED in 17.6s //tensorflow/compiler/tests:ftrl_ops_test_cpu_mlir_bridge_test PASSED in 24.7s //tensorflow/compiler/tests:ftrl_test_cpu PASSED in 26.4s //tensorflow/compiler/tests:function_test_cpu PASSED in 7.0s //tensorflow/compiler/tests:function_test_cpu_mlir_bridge_test PASSED in 8.2s //tensorflow/compiler/tests:gather_nd_op_test_cpu PASSED in 8.3s //tensorflow/compiler/tests:gather_nd_op_test_cpu_mlir_bridge_test PASSED in 7.6s //tensorflow/compiler/tests:gather_test_cpu PASSED in 38.6s //tensorflow/compiler/tests:gather_test_cpu_mlir_bridge_test PASSED in 44.7s //tensorflow/compiler/tests:jit_test_cpu PASSED in 50.0s //tensorflow/compiler/tests:listdiff_op_test_cpu PASSED in 12.5s //tensorflow/compiler/tests:listdiff_op_test_cpu_mlir_bridge_test PASSED in 13.2s //tensorflow/compiler/tests:lrn_ops_test_cpu PASSED in 7.8s //tensorflow/compiler/tests:lrn_ops_test_cpu_mlir_bridge_test PASSED in 7.2s //tensorflow/compiler/tests:lstm_test_cpu PASSED in 21.4s //tensorflow/compiler/tests:manip_ops_test_cpu PASSED in 11.0s //tensorflow/compiler/tests:manip_ops_test_cpu_mlir_bridge_test PASSED in 11.2s //tensorflow/compiler/tests:matrix_band_part_test_cpu PASSED in 43.8s //tensorflow/compiler/tests:matrix_band_part_test_cpu_mlir_bridge_test PASSED in 44.8s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu PASSED in 26.2s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu_mlir_bridge_test PASSED in 18.4s //tensorflow/compiler/tests:matrix_solve_op_test_cpu PASSED in 8.0s //tensorflow/compiler/tests:matrix_solve_op_test_cpu_mlir_bridge_test PASSED in 8.2s //tensorflow/compiler/tests:matrix_triangular_solve_op_test_cpu PASSED in 23.3s //tensorflow/compiler/tests:matrix_triangular_solve_op_test_cpu_mlir_bridge_test PASSED in 26.3s //tensorflow/compiler/tests:momentum_test_cpu PASSED in 10.7s //tensorflow/compiler/tests:nary_ops_test_cpu PASSED in 9.9s //tensorflow/compiler/tests:nary_ops_test_cpu_mlir_bridge_test PASSED in 8.7s //tensorflow/compiler/tests:nullary_ops_test_cpu PASSED in 10.1s //tensorflow/compiler/tests:nullary_ops_test_cpu_mlir_bridge_test PASSED in 7.7s //tensorflow/compiler/tests:placeholder_test_cpu PASSED in 8.8s //tensorflow/compiler/tests:placeholder_test_cpu_mlir_bridge_test PASSED in 9.9s //tensorflow/compiler/tests:proximal_adagrad_test_cpu PASSED in 8.3s //tensorflow/compiler/tests:proximal_gradient_descent_test_cpu PASSED in 9.2s //tensorflow/compiler/tests:quantized_ops_test_cpu PASSED in 8.2s //tensorflow/compiler/tests:reduce_window_test_cpu PASSED in 20.3s //tensorflow/compiler/tests:reduce_window_test_cpu_mlir_bridge_test PASSED in 8.5s //tensorflow/compiler/tests:reshape_op_test_cpu PASSED in 8.1s //tensorflow/compiler/tests:reshape_op_test_cpu_mlir_bridge_test PASSED in 7.7s //tensorflow/compiler/tests:reverse_ops_test_cpu PASSED in 14.2s //tensorflow/compiler/tests:reverse_ops_test_cpu_mlir_bridge_test PASSED in 11.0s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu PASSED in 7.8s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu_mlir_bridge_test PASSED in 8.9s //tensorflow/compiler/tests:risc_ops_test_cpu_mlir_bridge_test PASSED in 9.6s //tensorflow/compiler/tests:rmsprop_test_cpu PASSED in 11.5s //tensorflow/compiler/tests:scatter_nd_op_test_cpu PASSED in 19.3s //tensorflow/compiler/tests:scatter_nd_op_test_cpu_mlir_bridge_test PASSED in 26.1s //tensorflow/compiler/tests:searchsorted_op_test_cpu PASSED in 11.5s //tensorflow/compiler/tests:searchsorted_op_test_cpu_mlir_bridge_test PASSED in 9.2s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu PASSED in 24.5s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu_mlir_bridge_test PASSED in 27.2s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu PASSED in 18.7s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu_mlir_bridge_test PASSED in 17.5s //tensorflow/compiler/tests:slice_ops_test_cpu PASSED in 15.3s //tensorflow/compiler/tests:slice_ops_test_cpu_mlir_bridge_test PASSED in 36.2s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu PASSED in 9.0s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu_mlir_bridge_test PASSED in 8.6s //tensorflow/compiler/tests:stack_ops_test_cpu PASSED in 6.2s //tensorflow/compiler/tests:tensor_list_ops_test_cpu PASSED in 10.4s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu PASSED in 14.0s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu_mlir_bridge_test PASSED in 13.5s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu PASSED in 11.2s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu_mlir_bridge_test PASSED in 12.1s //tensorflow/compiler/tests:unique_ops_test_cpu PASSED in 6.9s //tensorflow/compiler/tests:variable_ops_test_cpu PASSED in 24.1s //tensorflow/compiler/tests:variable_ops_test_cpu_mlir_bridge_test PASSED in 16.6s //tensorflow/compiler/tests:where_op_test_cpu PASSED in 11.1s //tensorflow/compiler/tests:while_test_cpu PASSED in 9.2s //tensorflow/compiler/tests:xla_call_module_test_cpu PASSED in 8.3s //tensorflow/compiler/tests:xla_custom_call_ops_test_cpu PASSED in 7.5s //tensorflow/compiler/tests:xla_device_gpu_test_cpu PASSED in 7.8s //tensorflow/compiler/tests:xla_device_test_cpu PASSED in 12.1s //tensorflow/compiler/tests:xla_device_test_cpu_mlir_bridge_test PASSED in 13.4s //tensorflow/compiler/tests:xla_ops_test_cpu PASSED in 31.8s //tensorflow/compiler/tests:xla_ops_test_cpu_mlir_bridge_test PASSED in 38.7s //tensorflow/compiler/tests:xla_test_test PASSED in 6.9s //tensorflow/compiler/tf2xla:const_analysis_test PASSED in 5.0s //tensorflow/compiler/tf2xla:cpu_function_runtime_test PASSED in 0.1s //tensorflow/compiler/tf2xla:functionalize_cond_test PASSED in 0.6s //tensorflow/compiler/tf2xla:functionalize_control_flow_test PASSED in 0.6s //tensorflow/compiler/tf2xla:fused_batchnorm_reserve_space_test_cpu PASSED in 20.8s //tensorflow/compiler/tf2xla:literal_util_test PASSED in 1.0s //tensorflow/compiler/tf2xla:resource_operation_table_test PASSED in 5.7s //tensorflow/compiler/tf2xla:resource_util_test_cpu PASSED in 2.1s //tensorflow/compiler/tf2xla:sharding_util_test PASSED in 0.5s //tensorflow/compiler/tf2xla:tf2xla_test PASSED in 16.1s //tensorflow/compiler/tf2xla:tf2xla_util_test PASSED in 0.7s //tensorflow/compiler/tf2xla:xla_compiler_test PASSED in 15.5s //tensorflow/compiler/tf2xla:xla_jit_compiled_cpu_function_test PASSED in 15.9s //tensorflow/compiler/tf2xla:xla_op_registry_test PASSED in 4.6s //tensorflow/compiler/tf2xla/kernels:rng_converter_utils_test PASSED in 1.2s //tensorflow/compiler/xla:array2d_test PASSED in 0.3s //tensorflow/compiler/xla:array3d_test PASSED in 0.1s //tensorflow/compiler/xla:array4d_test PASSED in 0.9s //tensorflow/compiler/xla:array_test PASSED in 0.1s //tensorflow/compiler/xla:bit_cast_test PASSED in 0.2s //tensorflow/compiler/xla:comparison_util_test PASSED in 0.2s //tensorflow/compiler/xla:debug_options_parsers_test PASSED in 0.2s //tensorflow/compiler/xla:index_util_test PASSED in 0.1s //tensorflow/compiler/xla:iterator_util_test PASSED in 0.6s //tensorflow/compiler/xla:layout_test PASSED in 0.6s //tensorflow/compiler/xla:layout_util_test PASSED in 0.1s //tensorflow/compiler/xla:literal_test PASSED in 0.5s //tensorflow/compiler/xla:parse_flags_from_env_test PASSED in 0.4s //tensorflow/compiler/xla:permutation_util_test PASSED in 0.2s //tensorflow/compiler/xla:primitive_util_test PASSED in 0.2s //tensorflow/compiler/xla:refcounting_hash_map_test PASSED in 0.1s //tensorflow/compiler/xla:reference_util_test PASSED in 0.2s //tensorflow/compiler/xla:shape_test PASSED in 0.2s //tensorflow/compiler/xla:shape_tree_test PASSED in 0.1s //tensorflow/compiler/xla:shape_util_test PASSED in 6.4s //tensorflow/compiler/xla:status_macros_test PASSED in 0.2s //tensorflow/compiler/xla:text_literal_reader_test PASSED in 0.2s //tensorflow/compiler/xla:text_literal_writer_test PASSED in 1.3s //tensorflow/compiler/xla:types_test PASSED in 0.2s //tensorflow/compiler/xla:util_test PASSED in 0.6s //tensorflow/compiler/xla:window_util_test PASSED in 0.1s //tensorflow/compiler/xla/client:padding_test PASSED in 0.1s //tensorflow/compiler/xla/client:xla_builder_test PASSED in 0.3s //tensorflow/compiler/xla/client/lib:arithmetic_test_cpu PASSED in 7.0s //tensorflow/compiler/xla/client/lib:comparators_test_cpu PASSED in 7.0s //tensorflow/compiler/xla/client/lib:constants_test_cpu PASSED in 7.1s //tensorflow/compiler/xla/client/lib:logdet_test_cpu PASSED in 8.2s //tensorflow/compiler/xla/client/lib:math_test_cpu PASSED in 14.2s //tensorflow/compiler/xla/client/lib:matrix_test_cpu PASSED in 9.9s //tensorflow/compiler/xla/client/lib:pooling_test_cpu PASSED in 7.1s //tensorflow/compiler/xla/client/lib:qr_test_cpu PASSED in 12.5s //tensorflow/compiler/xla/client/lib:slicing_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/client/lib:sorting_test_cpu PASSED in 8.9s //tensorflow/compiler/xla/examples/axpy:stablehlo_compile_test PASSED in 9.2s //tensorflow/compiler/xla/experimental/conv_emitter:conv_emitter_test PASSED in 2.5s //tensorflow/compiler/xla/hlo/evaluator:hlo_evaluator_test PASSED in 2.0s //tensorflow/compiler/xla/hlo/transforms:hlo_constant_splitter_test PASSED in 0.8s //tensorflow/compiler/xla/hlo/utils:hlo_live_range_test PASSED in 1.3s //tensorflow/compiler/xla/hlo/utils:hlo_matchers_test PASSED in 1.5s //tensorflow/compiler/xla/hlo/utils:hlo_sharding_util_test PASSED in 0.2s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:collective_ops.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:collective_ops_to_cpu_runtime.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:fft.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:legalize_i1_vector_transfers.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:lmhlo_custom_call.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:lmhlo_infeed.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:remove_copies_to_out_params.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:rng_bit_generator.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_abi_legalization.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_memref_element_cast_to_llvm.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_outfeed.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_hlo_trace.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_launch.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memcpy.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memset.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_case.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_custom_call.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_fft.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cholesky.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_conv.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cublas_lt_matmul.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_gemm.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_infeed.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_outfeed.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_send_recv.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_while.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:memref_get_global_to_arg.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:outline_cuda_graphs.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/framework/tests:legalize-xla-framework.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/framework/tests:outline-with-xla-framework.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/framework/tests:xla-framework.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/math/transforms/tests:math_optimization.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/memref/transforms/tests:aligned_allocations.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops_verify.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir/runtime/ir/tests:testlib.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/runtime/transforms:calling_convention_test PASSED in 0.3s //tensorflow/compiler/xla/mlir/runtime/transforms:type_converter_test PASSED in 0.3s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:compilation_pipeline.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_asserts.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_custom_calls.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:export_functions.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:ordinal_assignment.mlir.test PASSED in 2.1s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:rt_to_llvm.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:erase-op-without-results.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:inline-scf-while.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:reduce-scf-forall-bounds.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-constant.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-value.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-operand-with-constant.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:return-operands-of-terminator-operands.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:truncate-function.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:bisect.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:no-bug.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:snapshot.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/tools/mlir_replay/public:execution_trace_utils_test PASSED in 0.1s //tensorflow/compiler/xla/mlir/utils:error_util_test PASSED in 0.1s //tensorflow/compiler/xla/mlir/xla_cpu/tests:bufferize.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/xla_cpu/tests:invalid.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/xla_cpu/tests:ops.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_hlo_broadcasts.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_hlo_no_broadcasts.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_mhlo.mlir.test PASSED in 2.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/sparse_chlo_legalize_to_linalg.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/buffer_reuse.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/canonicalize.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/convert_deallocation_ops_to_llvm.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocate.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_ops.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_to_scf.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/split_alloc_tensors.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/add_debug_info.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/bufferization.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/collapse-shape.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/collapse_forall_dimensions.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/compose_extract_insert_slice.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/conv_2d_nhwc_hwcf.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/dot.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/duplicate_fusions.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fibonacci.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fusion_outlining.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fusion_planning_for_cpu.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/inline_fusion_clusters.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_bcast_map.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_matmul.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_reduce.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_reduce_map.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/matmul.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_1d.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_2d.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_window.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reverse.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/scatter.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/sort.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/transpose.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/fusion.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/gml_st_to_scf.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/greedy_fusion.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/invalid.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/lower_vectors.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/nested_tiling.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/nested_tiling_softmax.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/ops.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/simplify_dead_copy.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/tile_by_one.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/tiling_and_fusion.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/tiling_softmax.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/vectorize_copy.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/vectorize_for_cpu.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-select-and-scatter.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-affine.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-gpu.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-parallel-loops.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-tensor-op.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/ops.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo_gpu/lhlo_gpu_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/attrs.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/broadcast_propagation.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/bitcast.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/canonicalize.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/concatenate.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/convert.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/convolution.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/custom_call.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/folder_limit.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reduce.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reshape.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reverse.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/scatter.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/transpose.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/tuple.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/while.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/constraint_fusion.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/convert_to_signless.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/expand_hlo_tuples.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/expand_ops_simplifier.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/group_reduction_dimensions.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-collapse-elementwise-map.mlir.test PASSED in 0.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-einsum-to-dot-general.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-gather-to-torch-index-select.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-shape-ops-to-standard.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-sort.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-arithmetic.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo-only-dynamic.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo-unranked.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-linalg.mlir.test PASSED in 3.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-memref-unranked.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-memref.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-stablehlo-experimental.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-stablehlo.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/inlining.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-control-flow.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-hlo-shape-computations.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-mhlo-to-thlo.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-to-std.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/lower-complex.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/lower-general-dot.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/materialize-broadcasts.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/merge_assuming_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_bytecode_customizations.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_dot.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_gather.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_reduction.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_scatter.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_flatten_tuple.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_infer_shape_type_methods.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_ops_prettyprint.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_reduce_pretty_print.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/ops.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/optimize-hlo.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/prepare-for-export.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/reify-result-types.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/restrict_max_rank.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/shape_legalize_to_hlo.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/shape_reification.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sink-constants-to-control-flow.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_gendot_lower.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_lower.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_ops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_rewriting.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_transpose.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/stablehlo-legalize-to-hlo.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/symbolic-shape-optimization.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/unfuse_batch_norm.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_bounds.mlir.test PASSED in 1.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_conv_op.mlir.test PASSED in 2.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_reduce_op.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_reduce_window_op.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_scatter_op.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_select_and_scatter_op.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_while_op.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/while_prettyprint.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/bufferize.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/canonicalize.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/legalize_sort.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/ops.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/tiling.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:alloc_to_arg.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:assuming-structural-propagation.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:buffer_packing.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:buffer_reuse.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:bufferize.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:bufferize_one_shot.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:collapse_parallel_loops_to_1d_pass.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:copy_removal.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:detensorize_scf_ops.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:index_type_llvm_lowering.mlir.test PASSED in 2.3s //tensorflow/compiler/xla/mlir_hlo/tests:legalize-trigonometric-to-approximation.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:lower_index_cast.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:propagate_static_shapes.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:rank-specialization.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:scalarization.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:shape-component-analysis.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:shape_simplification.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:test_userange.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:tile_loops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:unbufferize.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:unroll-loops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tools/mlir_interpreter/framework/tests:interpreter_value_test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tools/mlir_interpreter/framework/tests:tensor_or_memref_test PASSED in 0.1s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:binary.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:nullary.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:prepare-mhlo.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:ternary.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:unary.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/pjrt:host_callback_test PASSED in 0.5s //tensorflow/compiler/xla/pjrt:lru_cache_test PASSED in 0.1s //tensorflow/compiler/xla/pjrt:pjrt_api_test PASSED in 0.4s //tensorflow/compiler/xla/pjrt:pjrt_client_test_cpu PASSED in 7.0s //tensorflow/compiler/xla/pjrt:pjrt_compiler_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:pjrt_executable_test PASSED in 0.1s //tensorflow/compiler/xla/pjrt:pjrt_stream_executor_client_test PASSED in 7.7s //tensorflow/compiler/xla/pjrt:semaphore_test PASSED in 1.0s //tensorflow/compiler/xla/pjrt:tfrt_cpu_pjrt_client_test PASSED in 6.9s //tensorflow/compiler/xla/pjrt:tracked_device_buffer_test PASSED in 6.2s //tensorflow/compiler/xla/pjrt:tracked_tfrt_cpu_device_buffer_test PASSED in 0.1s //tensorflow/compiler/xla/pjrt:transpose_test PASSED in 44.1s //tensorflow/compiler/xla/pjrt/c:pjrt_c_api_cpu_test PASSED in 6.3s //tensorflow/compiler/xla/pjrt/c:pjrt_c_api_helpers_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt/distributed:service_test PASSED in 5.9s //tensorflow/compiler/xla/pjrt/gpu:se_gpu_pjrt_client_test PASSED in 2.3s //tensorflow/compiler/xla/python:outfeed_receiver_test_cpu PASSED in 7.9s //tensorflow/compiler/xla/python/ifrt:array_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:array_test_no_impl PASSED in 0.2s //tensorflow/compiler/xla/python/ifrt:client_test_no_impl PASSED in 0.5s //tensorflow/compiler/xla/python/ifrt:executable_test_no_impl PASSED in 0.8s //tensorflow/compiler/xla/python/ifrt:future_test PASSED in 0.2s //tensorflow/compiler/xla/python/ifrt:index_domain_test PASSED in 0.2s //tensorflow/compiler/xla/python/ifrt:index_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:shape_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:sharding_test PASSED in 0.9s //tensorflow/compiler/xla/python/ifrt:tuple_test_no_impl PASSED in 0.2s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_array_impl_test_tfrt_cpu PASSED in 12.1s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_client_impl_test_tfrt_cpu PASSED in 6.8s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_executable_impl_test_tfrt_cpu PASSED in 6.1s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_tuple_impl_test_tfrt_cpu PASSED in 6.1s //tensorflow/compiler/xla/python_api:xla_literal_test PASSED in 1.5s //tensorflow/compiler/xla/python_api:xla_shape_test PASSED in 0.9s //tensorflow/compiler/xla/rpc:grpc_client_test PASSED in 2.5s //tensorflow/compiler/xla/runtime:arguments_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:async_runtime_test PASSED in 0.1s //tensorflow/compiler/xla/runtime:custom_call_test PASSED in 1.9s //tensorflow/compiler/xla/runtime:diagnostics_test PASSED in 0.3s //tensorflow/compiler/xla/runtime:executable_test PASSED in 1.3s //tensorflow/compiler/xla/runtime:ffi_test PASSED in 1.0s //tensorflow/compiler/xla/runtime:map_by_type_test PASSED in 0.4s //tensorflow/compiler/xla/runtime:module_test PASSED in 0.1s //tensorflow/compiler/xla/runtime:results_test PASSED in 0.1s //tensorflow/compiler/xla/runtime:state_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:symbolic_shape_test PASSED in 0.1s //tensorflow/compiler/xla/runtime:type_id_test PASSED in 0.2s //tensorflow/compiler/xla/service:algebraic_simplifier_overflow_test_cpu PASSED in 7.2s //tensorflow/compiler/xla/service:algebraic_simplifier_test PASSED in 2.5s //tensorflow/compiler/xla/service:all_gather_broadcast_reorder_test PASSED in 1.1s //tensorflow/compiler/xla/service:all_gather_combiner_test PASSED in 1.1s //tensorflow/compiler/xla/service:all_gather_decomposer_test PASSED in 0.9s //tensorflow/compiler/xla/service:all_reduce_combiner_test PASSED in 0.9s //tensorflow/compiler/xla/service:all_reduce_contiguous_test PASSED in 1.0s //tensorflow/compiler/xla/service:all_reduce_folder_test PASSED in 1.1s //tensorflow/compiler/xla/service:all_reduce_promotion_test PASSED in 1.5s //tensorflow/compiler/xla/service:all_reduce_reassociate_test PASSED in 1.1s //tensorflow/compiler/xla/service:all_reduce_simplifier_test PASSED in 1.0s //tensorflow/compiler/xla/service:ar_crs_combiner_test PASSED in 1.1s //tensorflow/compiler/xla/service:async_collective_creator_test PASSED in 1.7s //tensorflow/compiler/xla/service:async_op_canonicalizer_test PASSED in 2.0s //tensorflow/compiler/xla/service:batch_dot_simplification_test PASSED in 0.9s //tensorflow/compiler/xla/service:batchnorm_expander_test_cpu PASSED in 7.5s //tensorflow/compiler/xla/service:bfloat16_conversion_folding_test PASSED in 1.3s //tensorflow/compiler/xla/service:bfloat16_propagation_test PASSED in 2.2s //tensorflow/compiler/xla/service:bitcast_dtypes_expander_test PASSED in 1.3s //tensorflow/compiler/xla/service:broadcast_canonicalizer_test PASSED in 2.4s //tensorflow/compiler/xla/service:buffer_assignment_test PASSED in 10.4s //tensorflow/compiler/xla/service:call_graph_test PASSED in 1.1s //tensorflow/compiler/xla/service:call_inliner_test PASSED in 2.0s //tensorflow/compiler/xla/service:change_op_data_type_test PASSED in 1.2s //tensorflow/compiler/xla/service:collective_ops_utils_test PASSED in 0.3s //tensorflow/compiler/xla/service:collectives_schedule_linearizer_test PASSED in 1.1s //tensorflow/compiler/xla/service:compilation_environments_test PASSED in 0.1s //tensorflow/compiler/xla/service:conditional_canonicalizer_test PASSED in 0.8s //tensorflow/compiler/xla/service:conditional_code_motion_test PASSED in 1.1s //tensorflow/compiler/xla/service:conditional_simplifier_test PASSED in 1.2s //tensorflow/compiler/xla/service:conditional_to_select_test PASSED in 1.1s //tensorflow/compiler/xla/service:convert_mover_test PASSED in 2.5s //tensorflow/compiler/xla/service:convert_operand_folding_test PASSED in 0.9s //tensorflow/compiler/xla/service:convolution_4d_expander_test PASSED in 0.9s //tensorflow/compiler/xla/service:convolution_group_converter_test PASSED in 1.0s //tensorflow/compiler/xla/service:convolution_pred_expander_test PASSED in 0.8s //tensorflow/compiler/xla/service:copy_insertion_test PASSED in 3.8s //tensorflow/compiler/xla/service:custom_call_status_test PASSED in 0.1s //tensorflow/compiler/xla/service:defuser_test PASSED in 0.7s //tensorflow/compiler/xla/service:despecializer_test PASSED in 1.0s //tensorflow/compiler/xla/service:dfs_hlo_visitor_with_default_test PASSED in 1.1s //tensorflow/compiler/xla/service:dot_decomposer_test PASSED in 1.1s //tensorflow/compiler/xla/service:dot_merger_test PASSED in 1.0s //tensorflow/compiler/xla/service:dynamic_dimension_inference_test PASSED in 1.1s //tensorflow/compiler/xla/service:dynamic_dimension_simplifier_test PASSED in 1.1s //tensorflow/compiler/xla/service:dynamic_index_splitter_test PASSED in 0.9s //tensorflow/compiler/xla/service:dynamic_padder_test_cpu PASSED in 16.6s //tensorflow/compiler/xla/service:dynamic_parameter_binding_test PASSED in 0.9s //tensorflow/compiler/xla/service:dynamic_update_slice_test_cpu PASSED in 18.5s //tensorflow/compiler/xla/service:elemental_ir_emitter_test_cpu PASSED in 11.1s //tensorflow/compiler/xla/service:flatten_call_graph_test PASSED in 0.9s //tensorflow/compiler/xla/service:float_normalization_test PASSED in 0.9s //tensorflow/compiler/xla/service:fusion_node_indexing_evaluation_test PASSED in 1.5s //tensorflow/compiler/xla/service:gather_expander_test PASSED in 5.9s //tensorflow/compiler/xla/service:gather_simplifier_test PASSED in 1.0s //tensorflow/compiler/xla/service:heap_simulator_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_activation_analysis_test PASSED in 1.4s //tensorflow/compiler/xla/service:hlo_alias_analysis_test PASSED in 1.4s //tensorflow/compiler/xla/service:hlo_casting_utils_test PASSED in 10.4s //tensorflow/compiler/xla/service:hlo_computation_deduplicator_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_computation_test PASSED in 2.2s //tensorflow/compiler/xla/service:hlo_constant_folding_test PASSED in 6.6s //tensorflow/compiler/xla/service:hlo_cost_analysis_test PASSED in 6.7s //tensorflow/compiler/xla/service:hlo_creation_utils_test PASSED in 4.2s //tensorflow/compiler/xla/service:hlo_cse_test PASSED in 9.3s //tensorflow/compiler/xla/service:hlo_dataflow_analysis_test PASSED in 1.6s //tensorflow/compiler/xla/service:hlo_dce_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_domain_test PASSED in 1.3s //tensorflow/compiler/xla/service:hlo_element_type_converter_test PASSED in 1.4s //tensorflow/compiler/xla/service:hlo_execution_profile_test PASSED in 7.0s //tensorflow/compiler/xla/service:hlo_graph_dumper_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_input_output_alias_config_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_instruction_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_liveness_analysis_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_memory_scheduler_test PASSED in 1.0s //tensorflow/compiler/xla/service:hlo_module_dce_test PASSED in 1.0s //tensorflow/compiler/xla/service:hlo_module_metadata_test PASSED in 0.2s //tensorflow/compiler/xla/service:hlo_module_test PASSED in 1.4s //tensorflow/compiler/xla/service:hlo_opcode_test PASSED in 0.3s //tensorflow/compiler/xla/service:hlo_ordering_test PASSED in 1.3s //tensorflow/compiler/xla/service:hlo_parser_test PASSED in 0.6s //tensorflow/compiler/xla/service:hlo_pass_pipeline_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_phi_graph_test PASSED in 0.2s //tensorflow/compiler/xla/service:hlo_proto_util_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_reachability_test PASSED in 1.0s //tensorflow/compiler/xla/service:hlo_rematerialization_test PASSED in 1.3s //tensorflow/compiler/xla/service:hlo_rematerialization_test_utils_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_replication_analysis_test PASSED in 1.6s //tensorflow/compiler/xla/service:hlo_schedule_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_sharding_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_verifier_test PASSED in 1.3s //tensorflow/compiler/xla/service:indexed_array_analysis_test PASSED in 1.2s //tensorflow/compiler/xla/service:instruction_fusion_test PASSED in 1.6s //tensorflow/compiler/xla/service:latency_hiding_scheduler_test PASSED in 1.2s //tensorflow/compiler/xla/service:layout_assignment_test PASSED in 6.3s //tensorflow/compiler/xla/service:layout_normalization_test PASSED in 2.1s //tensorflow/compiler/xla/service:logistic_expander_test PASSED in 1.0s //tensorflow/compiler/xla/service:loop_schedule_linearizer_test PASSED in 1.0s //tensorflow/compiler/xla/service:map_inliner_test PASSED in 1.0s //tensorflow/compiler/xla/service:mapped_ptr_container_sorter_test PASSED in 0.1s //tensorflow/compiler/xla/service:memory_space_assignment_best_fit_repacker_test PASSED in 0.4s //tensorflow/compiler/xla/service:memory_space_assignment_test PASSED in 1.3s //tensorflow/compiler/xla/service:memory_space_propagation_test PASSED in 0.7s //tensorflow/compiler/xla/service:name_uniquer_test PASSED in 0.3s //tensorflow/compiler/xla/service:operand_upcaster_test PASSED in 1.8s //tensorflow/compiler/xla/service:optimize_input_output_buffer_alias_test PASSED in 1.2s //tensorflow/compiler/xla/service:pattern_matcher_gmock_test PASSED in 0.2s //tensorflow/compiler/xla/service:pattern_matcher_test PASSED in 1.1s //tensorflow/compiler/xla/service:real_imag_expander_test PASSED in 0.9s //tensorflow/compiler/xla/service:reduce_decomposer_test PASSED in 1.2s //tensorflow/compiler/xla/service:reduce_scatter_combiner_test PASSED in 0.9s //tensorflow/compiler/xla/service:reduce_scatter_decomposer_test PASSED in 1.0s //tensorflow/compiler/xla/service:reduce_scatter_reassociate_test PASSED in 1.0s //tensorflow/compiler/xla/service:reshape_decomposer_test PASSED in 0.9s //tensorflow/compiler/xla/service:reshape_mover_test PASSED in 1.1s //tensorflow/compiler/xla/service:result_caster_test PASSED in 0.8s //tensorflow/compiler/xla/service:root_instruction_sinker_test PASSED in 1.2s //tensorflow/compiler/xla/service:scatter_expander_test PASSED in 1.1s //tensorflow/compiler/xla/service:scatter_simplifier_test PASSED in 1.2s //tensorflow/compiler/xla/service:select_and_scatter_expander_test PASSED in 1.6s //tensorflow/compiler/xla/service:shape_inference_test PASSED in 0.2s //tensorflow/compiler/xla/service:shaped_buffer_test PASSED in 6.0s //tensorflow/compiler/xla/service:sharding_propagation_test PASSED in 2.5s //tensorflow/compiler/xla/service:sharding_remover_test PASSED in 1.2s //tensorflow/compiler/xla/service:simplify_fp_conversions_test PASSED in 1.0s //tensorflow/compiler/xla/service:slice_sinker_test PASSED in 1.7s //tensorflow/compiler/xla/service:sort_simplifier_test PASSED in 0.9s //tensorflow/compiler/xla/service:space_to_batch_converter_test PASSED in 0.8s //tensorflow/compiler/xla/service:stable_sort_expander_test PASSED in 1.2s //tensorflow/compiler/xla/service:stochastic_convert_decomposer_test PASSED in 1.0s //tensorflow/compiler/xla/service:stream_pool_test PASSED in 0.1s //tensorflow/compiler/xla/service:topk_rewriter_test PASSED in 6.3s //tensorflow/compiler/xla/service:transpose_folding_test PASSED in 2.0s //tensorflow/compiler/xla/service:tuple_points_to_analysis_test PASSED in 1.3s //tensorflow/compiler/xla/service:tuple_simplifier_test PASSED in 1.0s //tensorflow/compiler/xla/service:tuple_util_test PASSED in 0.7s //tensorflow/compiler/xla/service:while_loop_all_reduce_code_motion_test PASSED in 1.1s //tensorflow/compiler/xla/service:while_loop_analysis_test PASSED in 1.2s //tensorflow/compiler/xla/service:while_loop_concat_code_motion_test PASSED in 0.9s //tensorflow/compiler/xla/service:while_loop_constant_sinking_test PASSED in 0.9s //tensorflow/compiler/xla/service:while_loop_expensive_invariant_code_motion_test PASSED in 1.1s //tensorflow/compiler/xla/service:while_loop_invariant_code_motion_test PASSED in 0.8s //tensorflow/compiler/xla/service:while_loop_simplifier_test PASSED in 1.1s //tensorflow/compiler/xla/service:while_loop_trip_count_annotator_test PASSED in 0.9s //tensorflow/compiler/xla/service:while_util_test PASSED in 0.8s //tensorflow/compiler/xla/service:xla_aot_compile_stablehlo_cpu_test PASSED in 7.5s //tensorflow/compiler/xla/service:xla_debug_info_manager_test PASSED in 0.9s //tensorflow/compiler/xla/service:zero_sized_hlo_elimination_test PASSED in 1.1s //tensorflow/compiler/xla/service/cpu:conv_canonicalization_test PASSED in 1.2s //tensorflow/compiler/xla/service/cpu:cpu_eigen_tensor_alignment_test PASSED in 1.1s //tensorflow/compiler/xla/service/cpu:cpu_instruction_fusion_test PASSED in 1.1s //tensorflow/compiler/xla/service/cpu:cpu_layout_assignment_test PASSED in 2.5s //tensorflow/compiler/xla/service/cpu:ir_emission_utils_test PASSED in 1.2s //tensorflow/compiler/xla/service/cpu:parallel_task_assignment_test PASSED in 3.0s //tensorflow/compiler/xla/service/cpu:runtime_fft_test PASSED in 0.2s //tensorflow/compiler/xla/service/cpu:shape_partition_test PASSED in 0.7s //tensorflow/compiler/xla/service/cpu:xfeed_manager_test PASSED in 0.7s //tensorflow/compiler/xla/service/cpu/tests:cpu_bytesizeof_test PASSED in 0.5s //tensorflow/compiler/xla/service/cpu/tests:cpu_dyn_shape_test PASSED in 7.5s //tensorflow/compiler/xla/service/cpu/tests:cpu_eigen_dot_operation_test PASSED in 6.5s //tensorflow/compiler/xla/service/cpu/tests:cpu_external_constants_test PASSED in 25.0s //tensorflow/compiler/xla/service/cpu/tests:cpu_fusion_test PASSED in 7.7s //tensorflow/compiler/xla/service/cpu/tests:cpu_infeed_test PASSED in 5.5s //tensorflow/compiler/xla/service/cpu/tests:cpu_intrinsic_test PASSED in 8.4s //tensorflow/compiler/xla/service/cpu/tests:cpu_key_value_sort_test PASSED in 6.4s //tensorflow/compiler/xla/service/cpu/tests:cpu_literal_caching_test PASSED in 7.4s //tensorflow/compiler/xla/service/cpu/tests:cpu_noalias_test PASSED in 8.1s //tensorflow/compiler/xla/service/cpu/tests:cpu_outfeed_test PASSED in 8.6s //tensorflow/compiler/xla/service/cpu/tests:cpu_profiling_test PASSED in 9.4s //tensorflow/compiler/xla/service/cpu/tests:cpu_spmd_compile_test PASSED in 7.0s //tensorflow/compiler/xla/service/cpu/tests:cpu_topk_test PASSED in 9.4s //tensorflow/compiler/xla/service/cpu/tests:cpu_vectorization_test PASSED in 9.1s //tensorflow/compiler/xla/service/cpu/tests:cpu_while_test PASSED in 8.6s //tensorflow/compiler/xla/service/cpu/tests:tree_reduction_rewriter_test PASSED in 8.7s //tensorflow/compiler/xla/service/gpu:alias_passthrough_params_test PASSED in 1.0s //tensorflow/compiler/xla/service/gpu:all_reduce_blueconnect_test PASSED in 1.1s //tensorflow/compiler/xla/service/gpu:cublas_pad_for_gemms_test PASSED in 1.7s //tensorflow/compiler/xla/service/gpu:cudnn_pad_for_convolutions_test PASSED in 2.2s //tensorflow/compiler/xla/service/gpu:cudnn_simplify_padding_test PASSED in 2.1s //tensorflow/compiler/xla/service/gpu:cudnn_support_utils_test PASSED in 1.1s //tensorflow/compiler/xla/service/gpu:cudnn_vectorize_convolutions_test PASSED in 1.3s //tensorflow/compiler/xla/service/gpu:fusion_merger_test PASSED in 1.8s //tensorflow/compiler/xla/service/gpu:gemm_rewriter_triton_test PASSED in 2.0s //tensorflow/compiler/xla/service/gpu:gpu_conv_padding_legalization_test PASSED in 1.3s //tensorflow/compiler/xla/service/gpu:gpu_conv_rewriter_test PASSED in 1.0s //tensorflow/compiler/xla/service/gpu:gpu_fusible_test PASSED in 2.5s //tensorflow/compiler/xla/service/gpu:gpu_hlo_cost_analysis_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu:gpu_performance_model_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu:gpu_sanitize_constant_names_test PASSED in 1.3s //tensorflow/compiler/xla/service/gpu:hlo_algorithm_denylist_test PASSED in 0.2s //tensorflow/compiler/xla/service/gpu:hlo_fusion_stats_test PASSED in 0.9s //tensorflow/compiler/xla/service/gpu:instruction_fusion_test PASSED in 2.4s //tensorflow/compiler/xla/service/gpu:ir_emission_utils_test PASSED in 1.7s //tensorflow/compiler/xla/service/gpu:matmul_utils_test PASSED in 0.8s //tensorflow/compiler/xla/service/gpu:move_copy_to_users_test PASSED in 1.5s //tensorflow/compiler/xla/service/gpu:multi_output_fusion_test PASSED in 1.9s //tensorflow/compiler/xla/service/gpu:non_atomically_upgradeable_rw_lock_test PASSED in 0.1s //tensorflow/compiler/xla/service/gpu:reduction_splitter_test PASSED in 1.5s //tensorflow/compiler/xla/service/gpu:scatter_slice_simplifier_test PASSED in 1.2s //tensorflow/compiler/xla/service/gpu:target_util_test PASSED in 0.5s //tensorflow/compiler/xla/service/gpu:variadic_op_splitter_test PASSED in 1.3s //tensorflow/compiler/xla/service/gpu:while_transformer_test PASSED in 1.3s //tensorflow/compiler/xla/service/gpu/llvm_gpu_backend:utils_test PASSED in 0.3s //tensorflow/compiler/xla/service/gpu/tests:gpu_reduce_scatter_creator_test PASSED in 0.9s //tensorflow/compiler/xla/service/gpu/tests:reduction_degenerate_dim_remover_test PASSED in 1.7s //tensorflow/compiler/xla/service/gpu/tests:reduction_dimension_grouper_test PASSED in 1.8s //tensorflow/compiler/xla/service/gpu/tests:tree_reduction_rewriter_test PASSED in 2.0s //tensorflow/compiler/xla/service/graphcycles:graphcycles_test PASSED in 0.7s //tensorflow/compiler/xla/service/graphcycles:ordered_set_test PASSED in 0.2s //tensorflow/compiler/xla/service/llvm_ir:alias_analysis_test PASSED in 6.0s //tensorflow/compiler/xla/service/llvm_ir:ir_array_test PASSED in 0.4s //tensorflow/compiler/xla/service/spmd:canonicalize_all_gather_for_cse_test PASSED in 0.9s //tensorflow/compiler/xla/service/spmd:collective_permute_motion_test PASSED in 0.9s //tensorflow/compiler/xla/service/spmd:partition_assignment_test PASSED in 1.0s //tensorflow/compiler/xla/service/spmd:schedule_aware_collective_ops_cse_test PASSED in 1.5s //tensorflow/compiler/xla/service/spmd:spmd_partitioner_test PASSED in 1.9s //tensorflow/compiler/xla/service/spmd:stateful_rng_spmd_partitioner_test PASSED in 1.0s //tensorflow/compiler/xla/stream_executor:dnn_test PASSED in 0.1s //tensorflow/compiler/xla/stream_executor:stream_test PASSED in 0.2s //tensorflow/compiler/xla/stream_executor/host:host_stream_test PASSED in 1.0s //tensorflow/compiler/xla/tests:all_reduce_test_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests:axpy_simple_test_cpu PASSED in 12.7s //tensorflow/compiler/xla/tests:bad_rng_shape_validation_test_cpu PASSED in 6.1s //tensorflow/compiler/xla/tests:binop_scaling_test_cpu PASSED in 6.3s //tensorflow/compiler/xla/tests:bitcast_convert_test_cpu PASSED in 12.9s //tensorflow/compiler/xla/tests:broadcast_simple_test_cpu PASSED in 8.8s //tensorflow/compiler/xla/tests:broadcast_test_cpu PASSED in 9.8s //tensorflow/compiler/xla/tests:buffer_donation_test_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests:call_test_cpu PASSED in 6.5s //tensorflow/compiler/xla/tests:check_execution_arity_test_cpu PASSED in 7.0s //tensorflow/compiler/xla/tests:cholesky_test_cpu PASSED in 14.0s //tensorflow/compiler/xla/tests:client_test_cpu PASSED in 8.5s //tensorflow/compiler/xla/tests:collective_ops_test_cpu PASSED in 18.4s //tensorflow/compiler/xla/tests:compilation_cache_test_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests:compute_constant_test_cpu PASSED in 6.7s //tensorflow/compiler/xla/tests:concat_test_cpu PASSED in 9.3s //tensorflow/compiler/xla/tests:constant_reduction_function_test_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests:constants_test_cpu PASSED in 7.0s //tensorflow/compiler/xla/tests:convert_test_cpu PASSED in 8.2s //tensorflow/compiler/xla/tests:copy_test_cpu PASSED in 8.3s //tensorflow/compiler/xla/tests:cpu_gpu_fusion_test_cpu PASSED in 9.7s //tensorflow/compiler/xla/tests:custom_call_test_cpu PASSED in 11.5s //tensorflow/compiler/xla/tests:deallocation_test_cpu PASSED in 8.4s //tensorflow/compiler/xla/tests:deconstruct_tuple_test_cpu PASSED in 10.5s //tensorflow/compiler/xla/tests:deep_graph_test_cpu PASSED in 6.5s //tensorflow/compiler/xla/tests:execution_profile_test_cpu PASSED in 6.4s //tensorflow/compiler/xla/tests:fft_test_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests:float8_test_cpu PASSED in 6.6s //tensorflow/compiler/xla/tests:floor_ceil_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests:fmax_fmin_test_cpu PASSED in 6.0s //tensorflow/compiler/xla/tests:gather_operation_test_cpu PASSED in 10.3s //tensorflow/compiler/xla/tests:get_dimension_size_test_cpu PASSED in 11.6s //tensorflow/compiler/xla/tests:half_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests:hlo_metadata_test PASSED in 5.7s //tensorflow/compiler/xla/tests:literal_test_util_test PASSED in 4.8s //tensorflow/compiler/xla/tests:local_client_allocation_test_cpu PASSED in 6.4s //tensorflow/compiler/xla/tests:local_client_aot_test PASSED in 0.1s //tensorflow/compiler/xla/tests:log_test_cpu PASSED in 6.5s //tensorflow/compiler/xla/tests:map_test_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests:matrix_ops_simple_test_cpu PASSED in 15.4s //tensorflow/compiler/xla/tests:multidimensional_slice_test_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests:multiple_devices_on_host_test PASSED in 9.2s //tensorflow/compiler/xla/tests:multithreaded_compilation_test_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests:outfeed_in_nested_computation_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests:pad_test_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests:pred_test_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests:query_inferred_shape_test_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests:reduce_hlo_test_cpu PASSED in 8.3s //tensorflow/compiler/xla/tests:reduce_precision_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests:replay_test_cpu PASSED in 6.5s //tensorflow/compiler/xla/tests:reshape_motion_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests:reverse_test_cpu PASSED in 6.7s //tensorflow/compiler/xla/tests:round_trip_packed_literal_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests:round_trip_transfer_test_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests:sample_text_test_cpu PASSED in 6.2s //tensorflow/compiler/xla/tests:scatter_test_cpu PASSED in 12.1s //tensorflow/compiler/xla/tests:select_test_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests:test_utils_test_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests:token_hlo_test_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests:transfer_manager_test_cpu PASSED in 13.6s //tensorflow/compiler/xla/tests:transpose_test_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests:tuple_test_cpu PASSED in 9.5s //tensorflow/compiler/xla/tests:unary_op_test_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests:value_inference_test_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests:vector_ops_reduce_test_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests:vector_ops_simple_test_cpu PASSED in 8.3s //tensorflow/compiler/xla/tests:while_test_cpu PASSED in 8.8s //tensorflow/compiler/xla/tools:hlo_control_flow_flattening_test PASSED in 5.8s //tensorflow/compiler/xla/tools:hlo_extractor_test PASSED in 1.0s //tensorflow/compiler/xla/tools:hlo_module_loader_test PASSED in 1.2s //tensorflow/compiler/xla/tools:interactive_graphviz_bin_test PASSED in 0.2s //tensorflow/compiler/xla/tools:run_hlo_module_bin_test PASSED in 0.7s //tensorflow/compiler/xla/tools/hlo_bisect:hlo_bisect_state_test PASSED in 2.0s //tensorflow/compiler/xla/translate/hlo_to_mhlo:hlo_utils_test PASSED in 1.0s //tensorflow/compiler/xla/translate/hlo_to_mhlo:mlir_hlo_builder_test PASSED in 0.6s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:bool_compare.hlotxt.test PASSED in 0.4s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:case_conditional.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:dynamic_param.hlo.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:entry_computation_layout.hlotxt.test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:frontend_attributes.hlotxt.test PASSED in 0.6s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fully_connected_reference_model.hlotxt.test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fusion.hlotxt.test PASSED in 0.9s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:if_conditional.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import.hlotxt.test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import_async.hlotxt.test PASSED in 0.6s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:layouts_and_names.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:location.hlotxt.test PASSED in 0.4s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:module_attributes.hlo.test PASSED in 1.2s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:simple.hlo.test PASSED in 0.6s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:spmd_module_sharding.hlo.test PASSED in 1.0s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:types.hlotxt.test PASSED in 0.4s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:while.hlotxt.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo:type_to_shape_test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:add.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:case.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:dynamic.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export-with-layouts.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export.mlir.test PASSED in 1.8s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_and_check_layouts.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_large_constants.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_replicas.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:frontend_attributes.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:fusion.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:if.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:input_output_aliasing.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:layouts_and_names.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_op_metadata.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:missing_main.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:module_attributes.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:multiple_return_tuple.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:opaque_elements_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:rng_get_and_update_state.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:sharding.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:simple.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:unsupported_type.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:while.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:hlo_text_to_lhlo_no_opt.hlotxt.test PASSED in 3.1s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:no_opt_ops.hlotxt.test PASSED in 0.9s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:non_identity_layouts.hlotxt.test PASSED in 1.2s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:ops.mlir.test PASSED in 2.1s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:passthrough.mlir.test PASSED in 0.7s //tensorflow/core:__tensorflow_core_lib_core_legacy_lib_core_all_tests PASSED in 8.4s //tensorflow/core:__tensorflow_core_lib_gtl_legacy_lib_gtl_tests PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_cell_reader_test PASSED in 37.4s //tensorflow/core:__tensorflow_core_lib_monitoring_collection_registry_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_counter_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_gauge_test PASSED in 0.4s //tensorflow/core:__tensorflow_core_lib_monitoring_metric_def_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_percentile_sampler_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_sampler_test PASSED in 0.3s //tensorflow/core:__tensorflow_core_lib_monitoring_test_utils_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_strings_legacy_low_level_library_tests PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_wav_wav_io_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_util_mkl_util_test_srcs PASSED in 0.2s //tensorflow/core:__tensorflow_tsl_lib_core_legacy_lib_core_all_tests PASSED in 0.4s //tensorflow/core:lib_strings_ordered_code_test PASSED in 1.7s //tensorflow/core:lib_strings_proto_serialization_test PASSED in 0.2s //tensorflow/core/api_def:api_test PASSED in 3.0s //tensorflow/core/api_def:update_api_def_test PASSED in 0.2s //tensorflow/core/common_runtime:all_to_all_test_cpu PASSED in 1.2s //tensorflow/core/common_runtime:arg_ret_placement_test PASSED in 0.6s //tensorflow/core/common_runtime:buf_rendezvous_test PASSED in 1.1s //tensorflow/core/common_runtime:collective_executor_mgr_test PASSED in 0.7s //tensorflow/core/common_runtime:collective_param_resolver_local_test PASSED in 5.1s //tensorflow/core/common_runtime:collective_rma_local_test PASSED in 0.9s //tensorflow/core/common_runtime:composite_device_test PASSED in 0.9s //tensorflow/core/common_runtime:cost_measurement_registry_test PASSED in 2.4s //tensorflow/core/common_runtime:cost_util_test PASSED in 0.2s //tensorflow/core/common_runtime:device_mgr_test PASSED in 1.3s //tensorflow/core/common_runtime:device_propagation_test PASSED in 0.7s //tensorflow/core/common_runtime:device_resolver_local_test PASSED in 0.7s //tensorflow/core/common_runtime:device_set_test PASSED in 0.7s //tensorflow/core/common_runtime:direct_session_test_cpu PASSED in 1.8s //tensorflow/core/common_runtime:direct_session_with_debug_test PASSED in 2.2s //tensorflow/core/common_runtime:direct_session_with_tracking_alloc_test PASSED in 1.0s //tensorflow/core/common_runtime:dynamic_device_mgr_test PASSED in 0.7s //tensorflow/core/common_runtime:eval_const_tensor_test PASSED in 0.9s //tensorflow/core/common_runtime:executor_test PASSED in 1.5s //tensorflow/core/common_runtime:function_optimization_registration_test PASSED in 0.8s //tensorflow/core/common_runtime:function_optimization_registry_no_pass_test PASSED in 1.9s //tensorflow/core/common_runtime:function_optimization_registry_pass_failure_test PASSED in 0.7s //tensorflow/core/common_runtime:function_optimization_registry_test PASSED in 0.8s //tensorflow/core/common_runtime:function_threadpool_test PASSED in 1.3s //tensorflow/core/common_runtime:graph_constructor_test PASSED in 1.9s //tensorflow/core/common_runtime:graph_runner_test PASSED in 1.1s //tensorflow/core/common_runtime:hierarchical_tree_broadcaster_test_cpu PASSED in 3.4s //tensorflow/core/common_runtime:inline_function_utils_test PASSED in 0.7s //tensorflow/core/common_runtime:input_colocation_exemption_registry_test PASSED in 0.7s //tensorflow/core/common_runtime:int32_fulltype_test PASSED in 0.5s //tensorflow/core/common_runtime:isolate_placer_inspection_required_ops_pass_test PASSED in 0.9s //tensorflow/core/common_runtime:lower_case_op_test PASSED in 2.2s //tensorflow/core/common_runtime:lower_function_call_test PASSED in 1.8s //tensorflow/core/common_runtime:lower_functional_ops_test PASSED in 2.3s //tensorflow/core/common_runtime:lower_if_op_test PASSED in 2.3s //tensorflow/core/common_runtime:lower_while_op_test PASSED in 2.1s //tensorflow/core/common_runtime:mkl_cpu_allocator_test PASSED in 0.2s //tensorflow/core/common_runtime:mkl_threadpool_device_test PASSED in 0.2s //tensorflow/core/common_runtime:no_op_cost_measurement_test PASSED in 0.1s //tensorflow/core/common_runtime:null_request_cost_accessor_test PASSED in 0.2s //tensorflow/core/common_runtime:optimization_registry_test PASSED in 0.7s //tensorflow/core/common_runtime:optimize_cross_host_control_deps_test PASSED in 13.8s //tensorflow/core/common_runtime:optimize_function_graph_utils_test PASSED in 0.9s //tensorflow/core/common_runtime:partitioning_utils_test PASSED in 1.1s //tensorflow/core/common_runtime:pending_counts_test PASSED in 1.2s //tensorflow/core/common_runtime:permuter_test_cpu PASSED in 3.2s //tensorflow/core/common_runtime:placer_inspection_required_ops_utils_test PASSED in 0.8s //tensorflow/core/common_runtime:placer_test PASSED in 1.2s //tensorflow/core/common_runtime:process_function_library_runtime_test_cpu PASSED in 0.8s //tensorflow/core/common_runtime:process_util_test PASSED in 0.1s //tensorflow/core/common_runtime:quantize_training_test PASSED in 1.9s //tensorflow/core/common_runtime:rendezvous_util_test PASSED in 0.1s //tensorflow/core/common_runtime:replicate_per_replica_nodes_test PASSED in 0.6s //tensorflow/core/common_runtime:request_cost_accessor_registry_test PASSED in 2.4s //tensorflow/core/common_runtime:request_cost_test PASSED in 0.1s //tensorflow/core/common_runtime:ring_gatherer_test_cpu PASSED in 2.0s //tensorflow/core/common_runtime:ring_reducer_test_cpu PASSED in 5.5s //tensorflow/core/common_runtime:scoped_allocator_mgr_test PASSED in 5.1s //tensorflow/core/common_runtime:session_test PASSED in 0.8s //tensorflow/core/common_runtime:shape_refiner_test PASSED in 0.8s //tensorflow/core/common_runtime:single_threaded_executor_test PASSED in 1.0s //tensorflow/core/common_runtime:threadpool_device_test PASSED in 0.7s //tensorflow/core/common_runtime:type_inference_test PASSED in 2.8s //tensorflow/core/common_runtime/eager:attr_builder_test PASSED in 25.2s //tensorflow/core/common_runtime/eager:context_test PASSED in 9.5s //tensorflow/core/common_runtime/eager:custom_device_test PASSED in 11.2s //tensorflow/core/common_runtime/eager:eager_executor_test PASSED in 9.6s //tensorflow/core/common_runtime/eager:eager_op_rewrite_registry_test PASSED in 1.9s //tensorflow/core/common_runtime/eager:eager_operation_test PASSED in 9.7s //tensorflow/core/common_runtime/eager:execute_node_test PASSED in 10.3s //tensorflow/core/common_runtime/eager:execute_test PASSED in 20.6s //tensorflow/core/common_runtime/eager:kernel_and_device_test PASSED in 0.9s //tensorflow/core/common_runtime/eager:mkl_eager_op_rewrite_test PASSED in 10.2s //tensorflow/core/common_runtime/eager:placement_test PASSED in 9.6s //tensorflow/core/common_runtime/eager:placement_utils_test PASSED in 10.7s //tensorflow/core/common_runtime/eager:tensor_handle_data_test PASSED in 8.5s //tensorflow/core/common_runtime/eager:tensor_handle_test PASSED in 11.2s //tensorflow/core/common_runtime/gpu:gpu_device_on_non_gpu_machine_test PASSED in 0.2s //tensorflow/core/common_runtime/next_pluggable_device/c:plugin_c_api_test PASSED in 4.5s //tensorflow/core/config:flags_py_test PASSED in 9.4s //tensorflow/core/config:flags_test PASSED in 0.1s //tensorflow/core/data:compression_utils_test PASSED in 2.5s //tensorflow/core/data:dataset_utils_test PASSED in 0.6s //tensorflow/core/data:hash_utils_test PASSED in 1.5s //tensorflow/core/data:metric_utils_test PASSED in 5.8s //tensorflow/core/data:name_utils_test PASSED in 0.1s //tensorflow/core/data:rewrite_utils_test PASSED in 1.1s //tensorflow/core/data:serialization_utils_test PASSED in 1.1s //tensorflow/core/data:snapshot_utils_test PASSED in 0.8s //tensorflow/core/data:split_utils_test PASSED in 0.5s //tensorflow/core/data:standalone_save_restore_test PASSED in 2.5s //tensorflow/core/data:standalone_test PASSED in 1.8s //tensorflow/core/data:tfdataz_metrics_test PASSED in 1.7s //tensorflow/core/data:unbounded_thread_pool_test PASSED in 0.9s //tensorflow/core/data/service:auto_shard_rewriter_test PASSED in 0.8s //tensorflow/core/data/service:common_test PASSED in 0.1s //tensorflow/core/data/service:credentials_factory_test PASSED in 0.5s //tensorflow/core/data/service:cross_trainer_cache_test PASSED in 1.6s //tensorflow/core/data/service:data_service_test PASSED in 12.4s //tensorflow/core/data/service:data_transfer_test PASSED in 0.5s //tensorflow/core/data/service:dataset_store_test PASSED in 0.6s //tensorflow/core/data/service:dispatcher_client_test PASSED in 3.0s //tensorflow/core/data/service:dispatcher_state_test PASSED in 0.6s //tensorflow/core/data/service:grpc_dispatcher_impl_test PASSED in 2.5s //tensorflow/core/data/service:grpc_util_test PASSED in 0.7s //tensorflow/core/data/service:grpc_worker_impl_test PASSED in 2.7s //tensorflow/core/data/service:journal_test PASSED in 0.5s //tensorflow/core/data/service:logging_utils_test PASSED in 0.2s //tensorflow/core/data/service:task_runner_test PASSED in 3.4s //tensorflow/core/data/service:test_util_test PASSED in 2.7s //tensorflow/core/data/service:url_test PASSED in 0.6s //tensorflow/core/data/service:utils_test PASSED in 1.2s //tensorflow/core/data/service:validate_utils_test PASSED in 0.1s //tensorflow/core/data/service:worker_client_test PASSED in 4.3s //tensorflow/core/data/service:worker_impl_test PASSED in 2.8s //tensorflow/core/data/service/client:data_service_client_test PASSED in 5.3s //tensorflow/core/data/service/client:utils_test PASSED in 3.1s //tensorflow/core/data/service/client:validate_utils_test PASSED in 1.1s //tensorflow/core/data/service/snapshot:distributed_snapshot_test PASSED in 19.4s //tensorflow/core/data/service/snapshot:file_utils_test PASSED in 0.7s //tensorflow/core/data/service/snapshot:path_utils_test PASSED in 0.2s //tensorflow/core/data/service/snapshot:snapshot_manager_test PASSED in 2.1s //tensorflow/core/data/service/snapshot:snapshot_reader_test PASSED in 2.2s //tensorflow/core/data/service/snapshot:snapshot_split_provider_test PASSED in 1.3s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_checkpoint_test PASSED in 3.2s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_test PASSED in 3.7s //tensorflow/core/data/service/snapshot:utils_test PASSED in 0.2s //tensorflow/core/debug:debug_graph_utils_test PASSED in 0.9s //tensorflow/core/distributed_runtime:call_options_test PASSED in 0.2s //tensorflow/core/distributed_runtime:cluster_function_library_runtime_test PASSED in 3.9s //tensorflow/core/distributed_runtime:collective_param_resolver_distributed_test PASSED in 1.0s //tensorflow/core/distributed_runtime:collective_rma_distributed_test PASSED in 0.7s //tensorflow/core/distributed_runtime:device_resolver_distributed_test PASSED in 0.6s //tensorflow/core/distributed_runtime:message_wrappers_test PASSED in 0.2s //tensorflow/core/distributed_runtime:partial_run_mgr_test PASSED in 0.4s //tensorflow/core/distributed_runtime:recent_request_ids_test PASSED in 0.3s //tensorflow/core/distributed_runtime:request_id_test PASSED in 0.3s //tensorflow/core/distributed_runtime:rpc_collective_executor_mgr_test PASSED in 0.4s //tensorflow/core/distributed_runtime:server_lib_test PASSED in 0.1s //tensorflow/core/distributed_runtime:session_mgr_test PASSED in 1.2s //tensorflow/core/distributed_runtime:tensor_coding_test PASSED in 0.3s //tensorflow/core/distributed_runtime/coordination:coordination_service_barrier_proxy_test PASSED in 2.3s //tensorflow/core/distributed_runtime/eager:eager_service_impl_test PASSED in 21.1s //tensorflow/core/distributed_runtime/eager:remote_mgr_test PASSED in 11.7s //tensorflow/core/distributed_runtime/integration_test:c_api_coordination_test_cpu PASSED in 46.1s //tensorflow/core/distributed_runtime/integration_test:c_api_multi_client_test_cpu PASSED in 27.0s //tensorflow/core/distributed_runtime/integration_test:c_api_recoverable_jobs_test_cpu PASSED in 39.5s //tensorflow/core/distributed_runtime/integration_test:c_api_session_coordination_test_cpu PASSED in 28.6s //tensorflow/core/distributed_runtime/rpc:grpc_tensor_coding_test PASSED in 2.3s //tensorflow/core/distributed_runtime/rpc:grpc_worker_cache_test PASSED in 0.9s //tensorflow/core/distributed_runtime/rpc/eager:grpc_eager_client_test PASSED in 0.6s //tensorflow/core/example:example_parser_configuration_test PASSED in 0.9s //tensorflow/core/example:feature_util_test PASSED in 0.1s //tensorflow/core/framework:allocator_test PASSED in 6.7s //tensorflow/core/framework:attr_value_util_test PASSED in 1.1s //tensorflow/core/framework:batch_util_test PASSED in 0.7s //tensorflow/core/framework:bfloat16_test PASSED in 0.8s //tensorflow/core/framework:common_shape_fns_test PASSED in 1.3s //tensorflow/core/framework:dataset_test PASSED in 0.9s //tensorflow/core/framework:device_base_test PASSED in 0.8s //tensorflow/core/framework:disable_jit_test PASSED in 1.1s //tensorflow/core/framework:framework_op_gen_lib_test PASSED in 0.1s //tensorflow/core/framework:framework_op_segment_test PASSED in 1.3s //tensorflow/core/framework:framework_resource_var_test PASSED in 0.3s //tensorflow/core/framework:framework_run_handler_test PASSED in 1.8s //tensorflow/core/framework:framework_run_handler_util_test PASSED in 2.3s //tensorflow/core/framework:full_type_inference_util_test PASSED in 1.3s //tensorflow/core/framework:full_type_util_test PASSED in 1.6s //tensorflow/core/framework:function_test PASSED in 2.3s //tensorflow/core/framework:graph_def_util_test PASSED in 0.9s //tensorflow/core/framework:graph_to_functiondef_test PASSED in 0.9s //tensorflow/core/framework:kernel_def_builder_test PASSED in 1.0s //tensorflow/core/framework:kernel_def_util_test PASSED in 1.3s //tensorflow/core/framework:memory_types_test PASSED in 0.9s //tensorflow/core/framework:model_test PASSED in 1.2s //tensorflow/core/framework:node_def_builder_test PASSED in 0.7s //tensorflow/core/framework:node_def_util_test PASSED in 1.3s //tensorflow/core/framework:node_properties_test PASSED in 1.8s //tensorflow/core/framework:op_compatibility_test PASSED in 1.8s //tensorflow/core/framework:op_def_builder_test PASSED in 1.3s //tensorflow/core/framework:op_def_util_test PASSED in 0.9s //tensorflow/core/framework:op_kernel_test PASSED in 0.7s //tensorflow/core/framework:op_registration_test PASSED in 0.9s //tensorflow/core/framework:partial_tensor_shape_test PASSED in 0.9s //tensorflow/core/framework:rendezvous_test PASSED in 3.0s //tensorflow/core/framework:resource_handle_test PASSED in 0.2s //tensorflow/core/framework:resource_mgr_test PASSED in 2.2s //tensorflow/core/framework:resource_op_kernel_test PASSED in 1.7s //tensorflow/core/framework:shape_inference_test PASSED in 1.2s //tensorflow/core/framework:shape_inference_testutil_test PASSED in 0.8s //tensorflow/core/framework:tensor_shape_test PASSED in 8.4s //tensorflow/core/framework:tensor_slice_test PASSED in 1.1s //tensorflow/core/framework:tensor_test PASSED in 29.9s //tensorflow/core/framework:tensor_testutil_test PASSED in 1.0s //tensorflow/core/framework:tensor_util_test PASSED in 0.6s //tensorflow/core/framework:tracking_allocator_test PASSED in 0.9s //tensorflow/core/framework:types_test PASSED in 1.2s //tensorflow/core/framework:variant_op_registry_test PASSED in 19.9s //tensorflow/core/framework:variant_test PASSED in 0.8s //tensorflow/core/framework/registration:registration_test PASSED in 0.7s //tensorflow/core/function/capture:by_ref_capture_test PASSED in 9.3s //tensorflow/core/function/capture:capture_container_test PASSED in 9.4s //tensorflow/core/function/integration_test:side_inputs_manual_api_test PASSED in 30.7s //tensorflow/core/function/integration_test:side_inputs_test PASSED in 14.6s //tensorflow/core/function/polymorphism:function_cache_test PASSED in 7.8s //tensorflow/core/function/polymorphism:function_type_test PASSED in 8.8s //tensorflow/core/function/polymorphism:type_dispatch_test PASSED in 6.5s //tensorflow/core/function/runtime_client:runtime_client_cc_test PASSED in 29.1s //tensorflow/core/function/trace_type:default_types_test PASSED in 9.9s //tensorflow/core/function/trace_type:serialization_test PASSED in 6.7s //tensorflow/core/function/trace_type:trace_type_test PASSED in 14.1s //tensorflow/core/graph:algorithm_test PASSED in 1.5s //tensorflow/core/graph:collective_order_test PASSED in 0.6s //tensorflow/core/graph:control_flow_test PASSED in 1.2s //tensorflow/core/graph:costmodel_test PASSED in 1.0s //tensorflow/core/graph:edgeset_test PASSED in 0.8s //tensorflow/core/graph:graph_def_builder_test PASSED in 0.7s //tensorflow/core/graph:graph_partition_test PASSED in 1.0s //tensorflow/core/graph:graph_test PASSED in 0.7s //tensorflow/core/graph:node_builder_test PASSED in 0.8s //tensorflow/core/graph:optimizer_cse_test PASSED in 0.8s //tensorflow/core/graph:subgraph_test PASSED in 0.9s //tensorflow/core/graph:tensor_id_test PASSED in 1.1s //tensorflow/core/graph:validate_test PASSED in 0.8s //tensorflow/core/graph/regularization:simple_delete_test PASSED in 0.4s //tensorflow/core/graph/regularization:util_test PASSED in 0.1s //tensorflow/core/grappler:graph_topology_view_test PASSED in 0.2s //tensorflow/core/grappler:graph_view_test PASSED in 1.6s //tensorflow/core/grappler:grappler_item_builder_test PASSED in 1.2s //tensorflow/core/grappler:grappler_item_test PASSED in 1.5s //tensorflow/core/grappler:mutable_graph_view_test PASSED in 1.2s //tensorflow/core/grappler:utils_test PASSED in 2.3s //tensorflow/core/grappler/clusters:single_machine_test PASSED in 23.8s //tensorflow/core/grappler/clusters:virtual_cluster_test PASSED in 0.9s //tensorflow/core/grappler/costs:analytical_cost_estimator_test PASSED in 2.0s //tensorflow/core/grappler/costs:cost_estimator_test PASSED in 0.2s //tensorflow/core/grappler/costs:graph_memory_test PASSED in 1.0s //tensorflow/core/grappler/costs:graph_properties_test PASSED in 4.8s //tensorflow/core/grappler/costs:robust_stats_test PASSED in 0.1s //tensorflow/core/grappler/costs:utils_test PASSED in 1.4s //tensorflow/core/grappler/costs:virtual_placer_test PASSED in 0.4s //tensorflow/core/grappler/costs:virtual_scheduler_test PASSED in 1.6s //tensorflow/core/grappler/graph_analyzer:gen_node_test PASSED in 1.6s //tensorflow/core/grappler/graph_analyzer:graph_analyzer_test PASSED in 1.6s //tensorflow/core/grappler/graph_analyzer:hash_tools_test PASSED in 1.4s //tensorflow/core/grappler/graph_analyzer:sig_node_test PASSED in 2.7s //tensorflow/core/grappler/graph_analyzer:subgraph_test PASSED in 1.5s //tensorflow/core/grappler/inputs:utils_test PASSED in 0.1s //tensorflow/core/grappler/optimizers:arithmetic_optimizer_test_cpu PASSED in 3.9s //tensorflow/core/grappler/optimizers:auto_mixed_precision_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:auto_parallel_test_cpu PASSED in 2.4s //tensorflow/core/grappler/optimizers:common_subgraph_elimination_test_cpu PASSED in 1.5s //tensorflow/core/grappler/optimizers:custom_graph_optimizer_registry_test_cpu PASSED in 4.8s //tensorflow/core/grappler/optimizers:debug_stripper_test_cpu PASSED in 1.8s //tensorflow/core/grappler/optimizers:dependency_optimizer_test_cpu PASSED in 1.7s //tensorflow/core/grappler/optimizers:evaluation_utils_test PASSED in 0.7s //tensorflow/core/grappler/optimizers:function_api_info_test PASSED in 0.2s //tensorflow/core/grappler/optimizers:function_optimizer_test_cpu PASSED in 3.2s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_factory_test PASSED in 0.3s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_test_cpu PASSED in 2.1s //tensorflow/core/grappler/optimizers:graph_optimizer_stage_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:implementation_selector_test PASSED in 1.3s //tensorflow/core/grappler/optimizers:loop_optimizer_test_cpu PASSED in 1.9s //tensorflow/core/grappler/optimizers:memory_optimizer_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:meta_optimizer_test_cpu PASSED in 7.0s //tensorflow/core/grappler/optimizers:mkl_remapper_test PASSED in 1.9s //tensorflow/core/grappler/optimizers:model_pruner_test_cpu PASSED in 1.8s //tensorflow/core/grappler/optimizers:pin_to_host_optimizer_test_cpu PASSED in 1.7s //tensorflow/core/grappler/optimizers:remapper_test_cpu PASSED in 2.0s //tensorflow/core/grappler/optimizers:scoped_allocator_optimizer_test PASSED in 2.6s //tensorflow/core/grappler/optimizers:shape_optimizer_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:static_schedule_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:tfg_optimizer_hook_test PASSED in 1.2s //tensorflow/core/grappler/optimizers/data:auto_shard_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:autotune_buffer_sizes_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:batch_parallelization_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:disable_intra_op_parallelism_test PASSED in 0.9s //tensorflow/core/grappler/optimizers/data:disable_prefetch_legacy_autotune_test PASSED in 0.9s //tensorflow/core/grappler/optimizers/data:enable_gradient_descent_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:filter_fusion_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:filter_parallelization_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:function_utils_test PASSED in 1.1s //tensorflow/core/grappler/optimizers/data:fusion_utils_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:graph_utils_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:inject_prefetch_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:make_deterministic_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:make_sloppy_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:map_and_batch_fusion_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:map_and_filter_fusion_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:map_fusion_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:map_parallelization_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:noop_elimination_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:parallel_batch_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:replicate_on_split_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:shuffle_and_repeat_fusion_test PASSED in 0.3s //tensorflow/core/grappler/optimizers/data:slack_test PASSED in 1.3s //tensorflow/core/grappler/optimizers/data:split_utils_test PASSED in 1.0s //tensorflow/core/grappler/optimizers/data:use_private_thread_pool_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/inference:batch_op_rewriter_test PASSED in 1.0s //tensorflow/core/grappler/utils:canonicalizer_test PASSED in 1.1s //tensorflow/core/grappler/utils:colocation_test PASSED in 0.4s //tensorflow/core/grappler/utils:frame_test PASSED in 0.3s //tensorflow/core/grappler/utils:functions_test PASSED in 1.4s //tensorflow/core/grappler/utils:graph_view_internal_test PASSED in 0.4s //tensorflow/core/grappler/utils:graph_view_test PASSED in 2.2s //tensorflow/core/grappler/utils:grappler_test_test PASSED in 7.3s //tensorflow/core/grappler/utils:pattern_utils_test PASSED in 0.7s //tensorflow/core/grappler/utils:scc_test PASSED in 1.7s //tensorflow/core/grappler/utils:symbolic_shapes_test PASSED in 0.2s //tensorflow/core/grappler/utils:topological_sort_test PASSED in 0.5s //tensorflow/core/grappler/utils:tpu_test PASSED in 0.2s //tensorflow/core/grappler/utils:transitive_fanin_test PASSED in 0.6s //tensorflow/core/grappler/utils:traversal_test PASSED in 0.6s //tensorflow/core/grappler/verifiers:structure_verifier_test PASSED in 1.5s //tensorflow/core/ir:interfaces_test PASSED in 0.3s //tensorflow/core/ir:ops_test PASSED in 1.1s //tensorflow/core/ir:shape_inference_utils_test PASSED in 0.3s //tensorflow/core/ir:tf_op_registry_test PASSED in 1.0s //tensorflow/core/ir:tf_op_wrapper_test PASSED in 0.2s //tensorflow/core/ir:utility_test PASSED in 0.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:arg_as_control_ret.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:backedge_segment.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:empty.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:error_during_backedge.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_case_with_attr_inference.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_if_with_attr_inference.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_iterator_get_next_attr_inference.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_underscore_output_shapes.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_while_with_attr_inference.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infeed_dequeue.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_arg_handle_type.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_with_output_shapes.pbtxt.test PASSED in 0.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_arg_name.pbtxt.test PASSED in 0.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_backedge_input_size.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_duplicated_node_name.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_index.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_name.pbtxt.test PASSED in 0.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_attr_key.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_key.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_name.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_op_type.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_func_with_empty_name.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_function_import.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_control_result.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_input.pbtxt.test PASSED in 1.3s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_name.pbtxt.test PASSED in 1.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_result.pbtxt.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_attr_name.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_named_edge_index.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_handle_data.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_input.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result_value.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result_value.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_input.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_two_inputs.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_named_edge_index.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_op_name.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_type_list.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:legacy_call.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_shape.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_zero_constant.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:three_nodes_with_attrs.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:version.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:empty.mlir.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:fulltype.mlir.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:func_with_no_args_or_results.mlir.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:negative_zero_constant.mlir.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:nested_legacy_call.mlir.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:three_nodes_with_attrs.mlir.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:version.mlir.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/saved_model:saved_model_roundtrip_test PASSED in 0.3s //tensorflow/core/ir/tests:attributes.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:canonicalize.mlir.test PASSED in 0.9s //tensorflow/core/ir/tests:compatible_types.mlir.test PASSED in 0.4s //tensorflow/core/ir/tests:concrete-ops.mlir.test PASSED in 1.1s //tensorflow/core/ir/tests:generic_concrete_ops.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:invalid-concrete-ops.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:invalid-preserved-attrs.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:invalid.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:invalid_types.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:ops.mlir.test PASSED in 5.7s //tensorflow/core/ir/tests:region-invalid-ops.mlir.test PASSED in 0.4s //tensorflow/core/ir/tests:region-ops-graph.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:region-ops.mlir.test PASSED in 0.8s //tensorflow/core/ir/tests:types.mlir.test PASSED in 0.6s //tensorflow/core/ir/types:dialect_test PASSED in 0.2s //tensorflow/core/kernels:as_string_op_test PASSED in 0.7s //tensorflow/core/kernels:basic_ops_benchmark_test PASSED in 0.5s //tensorflow/core/kernels:batch_kernels_env_test PASSED in 0.7s //tensorflow/core/kernels:batch_kernels_test PASSED in 1.2s //tensorflow/core/kernels:bias_op_test PASSED in 0.7s //tensorflow/core/kernels:bincount_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:broadcast_to_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:cast_op_test_cpu PASSED in 1.9s //tensorflow/core/kernels:checkpoint_callback_manager_test PASSED in 0.4s //tensorflow/core/kernels:clustering_ops_test PASSED in 0.4s //tensorflow/core/kernels:composite_tensor_variant_test PASSED in 0.5s //tensorflow/core/kernels:concat_op_test PASSED in 0.5s //tensorflow/core/kernels:constant_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:control_flow_ops_test PASSED in 7.4s //tensorflow/core/kernels:conv_grad_filter_ops_benchmark_test_cpu PASSED in 1.0s //tensorflow/core/kernels:conv_grad_input_ops_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels:conv_ops_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels:conv_ops_test_cpu PASSED in 6.1s //tensorflow/core/kernels:count_ops_test PASSED in 0.7s //tensorflow/core/kernels:cross_op_test PASSED in 0.9s //tensorflow/core/kernels:cwise_ops_test_cpu PASSED in 0.7s //tensorflow/core/kernels:debug_ops_test PASSED in 0.9s //tensorflow/core/kernels:decode_wav_op_test PASSED in 2.0s //tensorflow/core/kernels:deep_conv2d_test PASSED in 0.7s //tensorflow/core/kernels:dequantize_op_test PASSED in 0.7s //tensorflow/core/kernels:diag_op_test_cpu PASSED in 1.6s //tensorflow/core/kernels:dynamic_partition_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:dynamic_stitch_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:eigen_activations_test PASSED in 0.2s //tensorflow/core/kernels:eigen_attention_test PASSED in 0.1s //tensorflow/core/kernels:eigen_backward_cuboid_convolutions_test PASSED in 1.4s //tensorflow/core/kernels:eigen_backward_spatial_convolutions_test PASSED in 0.2s //tensorflow/core/kernels:eigen_benchmark_cpu_test PASSED in 1.1s //tensorflow/core/kernels:eigen_mkldnn_contraction_kernel_test PASSED in 0.2s //tensorflow/core/kernels:eigen_pooling_test PASSED in 0.5s //tensorflow/core/kernels:encode_wav_op_test PASSED in 1.5s //tensorflow/core/kernels:fingerprint_op_test PASSED in 0.5s //tensorflow/core/kernels:fused_batch_norm_ex_op_test_cpu PASSED in 1.1s //tensorflow/core/kernels:fused_batch_norm_op_test_cpu PASSED in 1.5s //tensorflow/core/kernels:gather_nd_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:gather_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:guarantee_const_op_test PASSED in 1.2s //tensorflow/core/kernels:identity_n_op_test PASSED in 1.1s //tensorflow/core/kernels:identity_op_test PASSED in 1.4s //tensorflow/core/kernels:immutable_constant_op_test PASSED in 0.8s //tensorflow/core/kernels:in_topk_op_test PASSED in 0.7s //tensorflow/core/kernels:isotonic_regression_op_test PASSED in 1.1s //tensorflow/core/kernels:logging_ops_test PASSED in 1.9s //tensorflow/core/kernels:lookup_ops_test PASSED in 1.1s //tensorflow/core/kernels:loss_test PASSED in 0.4s //tensorflow/core/kernels:lrn_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:matmul_op_test_cpu PASSED in 2.7s //tensorflow/core/kernels:merge_v2_checkpoints_op_test PASSED in 0.9s //tensorflow/core/kernels:mfcc_dct_test PASSED in 0.2s //tensorflow/core/kernels:mfcc_mel_filterbank_test PASSED in 0.1s //tensorflow/core/kernels:mfcc_op_test_cpu PASSED in 2.7s //tensorflow/core/kernels:mfcc_test PASSED in 0.2s //tensorflow/core/kernels:multinomial_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:nn_ops_test_cpu PASSED in 0.5s //tensorflow/core/kernels:one_hot_op_test PASSED in 0.4s //tensorflow/core/kernels:ops_testutil_test PASSED in 1.1s //tensorflow/core/kernels:ops_util_test PASSED in 0.1s //tensorflow/core/kernels:parameterized_truncated_normal_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:parse_tensor_test PASSED in 0.8s //tensorflow/core/kernels:quantization_utils_test PASSED in 0.7s //tensorflow/core/kernels:quantize_and_dequantize_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:quantize_down_and_shrink_range_op_test PASSED in 0.6s //tensorflow/core/kernels:quantize_op_test PASSED in 1.3s //tensorflow/core/kernels:quantized_activation_ops_test PASSED in 0.7s //tensorflow/core/kernels:quantized_add_op_test PASSED in 1.1s //tensorflow/core/kernels:quantized_batch_norm_op_test PASSED in 0.7s //tensorflow/core/kernels:quantized_bias_add_op_test PASSED in 0.8s //tensorflow/core/kernels:quantized_concat_op_test PASSED in 0.8s //tensorflow/core/kernels:quantized_conv_ops_test PASSED in 0.6s //tensorflow/core/kernels:quantized_instance_norm_test PASSED in 1.1s //tensorflow/core/kernels:quantized_matmul_op_test PASSED in 0.8s //tensorflow/core/kernels:quantized_mul_op_test PASSED in 0.9s //tensorflow/core/kernels:quantized_pooling_ops_test PASSED in 0.6s //tensorflow/core/kernels:quantized_reshape_op_test PASSED in 0.6s //tensorflow/core/kernels:quantized_resize_bilinear_op_test PASSED in 1.9s //tensorflow/core/kernels:ragged_fill_empty_rows_op_test PASSED in 0.5s //tensorflow/core/kernels:ragged_gather_op_test PASSED in 1.0s //tensorflow/core/kernels:ragged_range_op_test PASSED in 1.3s //tensorflow/core/kernels:ragged_tensor_from_variant_op_test PASSED in 0.8s //tensorflow/core/kernels:ragged_tensor_to_sparse_kernel_test PASSED in 0.8s //tensorflow/core/kernels:ragged_tensor_to_tensor_op_test PASSED in 0.7s //tensorflow/core/kernels:ragged_tensor_to_variant_op_test PASSED in 0.8s //tensorflow/core/kernels:random_binomial_op_test_cpu PASSED in 1.0s //tensorflow/core/kernels:random_index_shuffle_test PASSED in 0.3s //tensorflow/core/kernels:random_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:random_poisson_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels:range_sampler_test PASSED in 0.3s //tensorflow/core/kernels:reduction_ops_test_cpu PASSED in 0.9s //tensorflow/core/kernels:regex_replace_op_test PASSED in 0.7s //tensorflow/core/kernels:requantization_range_op_test PASSED in 1.6s //tensorflow/core/kernels:requantize_op_test PASSED in 0.8s //tensorflow/core/kernels:resource_ops_test PASSED in 0.9s //tensorflow/core/kernels:restore_op_test PASSED in 0.5s //tensorflow/core/kernels:restore_v2_op_test PASSED in 5.5s //tensorflow/core/kernels:reverse_op_test PASSED in 1.2s //tensorflow/core/kernels:roll_op_test PASSED in 0.9s //tensorflow/core/kernels:save_op_test PASSED in 0.7s //tensorflow/core/kernels:save_v2_op_test PASSED in 0.6s //tensorflow/core/kernels:scan_ops_test_cpu PASSED in 0.9s //tensorflow/core/kernels:scatter_nd_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:scatter_op_test PASSED in 1.0s //tensorflow/core/kernels:scoped_allocator_ops_test_cpu PASSED in 8.4s //tensorflow/core/kernels:sdca_ops_test PASSED in 1.9s //tensorflow/core/kernels:segment_reduction_ops_test PASSED in 0.7s //tensorflow/core/kernels:sendrecv_ops_test PASSED in 0.8s //tensorflow/core/kernels:sequence_ops_test PASSED in 0.7s //tensorflow/core/kernels:shape_ops_test PASSED in 0.7s //tensorflow/core/kernels:slice_op_test PASSED in 0.7s //tensorflow/core/kernels:spacetobatch_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels:sparse_add_op_test PASSED in 0.4s //tensorflow/core/kernels:sparse_dense_binary_op_shared_test PASSED in 0.7s //tensorflow/core/kernels:sparse_fill_empty_rows_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:sparse_matmul_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels:sparse_reduce_sum_op_test PASSED in 1.3s //tensorflow/core/kernels:sparse_tensor_dense_matmul_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels:sparse_to_dense_op_test_cpu PASSED in 1.0s //tensorflow/core/kernels:sparse_utils_test PASSED in 0.4s //tensorflow/core/kernels:sparse_xent_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:spectrogram_op_test_cpu PASSED in 2.8s //tensorflow/core/kernels:spectrogram_test PASSED in 0.1s //tensorflow/core/kernels:split_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:split_v_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:strided_slice_op_test PASSED in 0.8s //tensorflow/core/kernels:string_format_op_test PASSED in 0.6s //tensorflow/core/kernels:string_ngrams_op_test PASSED in 0.9s //tensorflow/core/kernels:string_split_op_test PASSED in 0.4s //tensorflow/core/kernels:substr_op_test PASSED in 0.6s //tensorflow/core/kernels:summary_audio_op_test PASSED in 0.7s //tensorflow/core/kernels:summary_image_op_test PASSED in 0.5s //tensorflow/core/kernels:summary_op_test PASSED in 0.5s //tensorflow/core/kernels:summary_tensor_op_test PASSED in 0.9s //tensorflow/core/kernels:tensor_cord_test PASSED in 0.2s //tensorflow/core/kernels:tensor_flag_utils_test PASSED in 0.1s //tensorflow/core/kernels:tensor_map_test PASSED in 0.1s //tensorflow/core/kernels:training_ops_test PASSED in 0.6s //tensorflow/core/kernels:transpose_util_test PASSED in 0.6s //tensorflow/core/kernels:unary_ops_composition_test_cpu PASSED in 2.7s //tensorflow/core/kernels:unique_op_test PASSED in 0.6s //tensorflow/core/kernels:variable_ops_test PASSED in 1.4s //tensorflow/core/kernels:while_op_test PASSED in 1.1s //tensorflow/core/kernels:xent_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels/batching_util:basic_batch_scheduler_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:batch_input_task_test PASSED in 0.5s //tensorflow/core/kernels/batching_util:batch_resource_base_test PASSED in 0.3s //tensorflow/core/kernels/batching_util:batch_scheduler_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:bounded_executor_test PASSED in 20.2s //tensorflow/core/kernels/batching_util:input_split_metadata_test PASSED in 0.5s //tensorflow/core/kernels/batching_util:periodic_function_test PASSED in 2.2s //tensorflow/core/kernels/batching_util:serial_device_batch_scheduler_test PASSED in 1.6s //tensorflow/core/kernels/batching_util:shared_batch_scheduler_test PASSED in 3.4s //tensorflow/core/kernels/batching_util:threadsafe_status_test PASSED in 0.1s //tensorflow/core/kernels/data:batch_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:cache_dataset_ops_test PASSED in 0.9s //tensorflow/core/kernels/data:concatenate_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data:filter_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:finalize_dataset_op_test PASSED in 1.7s //tensorflow/core/kernels/data:fixed_length_record_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:flat_map_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:get_options_op_test PASSED in 0.6s //tensorflow/core/kernels/data:interleave_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data:iterator_ops_test PASSED in 0.9s //tensorflow/core/kernels/data:map_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:map_defun_op_test PASSED in 0.5s //tensorflow/core/kernels/data:optimize_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:options_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:padded_batch_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:parallel_batch_dataset_op_test PASSED in 2.0s //tensorflow/core/kernels/data:parallel_filter_dataset_op_test PASSED in 1.8s //tensorflow/core/kernels/data:parallel_interleave_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:parallel_map_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:prefetch_autotuner_test PASSED in 0.2s //tensorflow/core/kernels/data:prefetch_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:range_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:reduce_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:repeat_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:rewrite_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:shard_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:shuffle_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:skip_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:sparse_tensor_slice_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:take_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data:tensor_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:tensor_slice_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:text_line_dataset_op_test PASSED in 1.4s //tensorflow/core/kernels/data:tf_record_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:window_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:zip_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data/experimental:assert_next_dataset_op_test PASSED in 1.4s //tensorflow/core/kernels/data/experimental:assert_prev_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data/experimental:auto_shard_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data/experimental:directed_interleave_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data/experimental:list_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data/experimental:map_and_batch_dataset_op_test PASSED in 1.6s //tensorflow/core/kernels/data/experimental:parallel_interleave_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data/experimental:random_dataset_op_test PASSED in 1.6s //tensorflow/core/kernels/data/experimental:sampling_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data/experimental:save_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data/experimental:unique_dataset_op_test PASSED in 1.7s //tensorflow/core/kernels/image:adjust_contrast_op_benchmark_test_cpu PASSED in 0.7s //tensorflow/core/kernels/image:adjust_contrast_op_test PASSED in 0.7s //tensorflow/core/kernels/image:colorspace_op_test PASSED in 1.1s //tensorflow/core/kernels/image:crop_and_resize_op_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels/image:crop_and_resize_op_test PASSED in 0.8s //tensorflow/core/kernels/image:encode_jpeg_op_test PASSED in 0.6s //tensorflow/core/kernels/image:mirror_pad_op_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels/image:mirror_pad_op_test PASSED in 0.8s //tensorflow/core/kernels/image:non_max_suppression_op_benchmark_test PASSED in 0.6s //tensorflow/core/kernels/image:non_max_suppression_op_test PASSED in 1.3s //tensorflow/core/kernels/image:resize_area_op_test PASSED in 1.0s //tensorflow/core/kernels/image:resize_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels/image:resize_bicubic_op_test PASSED in 4.3s //tensorflow/core/kernels/image:resize_ops_test_cpu PASSED in 2.5s //tensorflow/core/kernels/image:sampling_kernels_test PASSED in 0.4s //tensorflow/core/kernels/image:scale_and_translate_op_test PASSED in 2.0s //tensorflow/core/kernels/linalg:banded_triangular_solve_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels/linalg:matrix_triangular_solve_op_test_cpu PASSED in 2.1s //tensorflow/core/kernels/mkl:mkl_conv_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_dequantize_op_test PASSED in 0.3s //tensorflow/core/kernels/mkl:mkl_fused_batch_norm_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_fused_ops_test PASSED in 0.3s //tensorflow/core/kernels/mkl:mkl_matmul_op_benchmark PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_qmatmul_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantize_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantized_concat_op_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_perchannel_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_quantized_pooling_ops_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_relu_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_requantize_ops_test PASSED in 0.3s //tensorflow/core/kernels/mkl:mkl_swish_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:onednn_nn_ops_benchmark PASSED in 0.2s //tensorflow/core/kernels/sparse:kernels_test PASSED in 0.9s //tensorflow/core/kernels/uniform_quant_ops:math_utils_test PASSED in 0.7s //tensorflow/core/kernels/uniform_quant_ops:tensor_utils_test PASSED in 0.5s //tensorflow/core/kernels/uniform_quant_ops:uniform_dequantize_op_test PASSED in 0.5s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantize_op_test PASSED in 0.6s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_add_op_test PASSED in 0.5s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_clip_by_value_op_test PASSED in 0.6s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_convolution_ops_test PASSED in 1.0s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_dot_ops_test PASSED in 0.6s //tensorflow/core/kernels/uniform_quant_ops:uniform_requantize_op_test PASSED in 0.6s //tensorflow/core/lib/db:sqlite_test PASSED in 0.3s //tensorflow/core/lib/gif:lib_gif_io_test PASSED in 1.9s //tensorflow/core/lib/jpeg:lib_jpeg_jpeg_mem_unittest PASSED in 0.8s //tensorflow/core/ops:cudnn_rnn_ops_test_cc PASSED in 0.5s //tensorflow/core/ops:ops_array_grad_test PASSED in 1.1s //tensorflow/core/ops:ops_math_grad_test PASSED in 3.7s //tensorflow/core/ops:ops_tests PASSED in 0.8s //tensorflow/core/ops/compat:backwards_compatibility_test PASSED in 0.5s //tensorflow/core/platform:__tensorflow_tsl_platform_profile_utils_cpu_utils_test PASSED in 0.1s //tensorflow/core/platform:enable_tf2_utils_test PASSED in 0.2s //tensorflow/core/platform:env_test PASSED in 3.3s //tensorflow/core/platform:fake_python_env_test PASSED in 0.1s //tensorflow/core/platform:file_system_test PASSED in 0.3s //tensorflow/core/platform:platform_strings_test PASSED in 0.1s //tensorflow/core/platform:ram_file_system_test PASSED in 30.9s //tensorflow/core/platform:resource_loader_test PASSED in 0.3s //tensorflow/core/platform:vmodule_benchmark_test PASSED in 0.1s //tensorflow/core/platform:vmodule_test PASSED in 0.2s //tensorflow/core/profiler/backends/cpu:host_tracer_test PASSED in 0.1s //tensorflow/core/profiler/convert:hlo_proto_to_graph_view_test PASSED in 0.2s //tensorflow/core/profiler/convert:hlo_proto_to_memory_visualization_utils_test PASSED in 0.3s //tensorflow/core/profiler/convert:op_stats_to_pod_stats_test PASSED in 0.2s //tensorflow/core/profiler/convert:op_stats_to_pod_viewer_test PASSED in 0.3s //tensorflow/core/profiler/convert:op_stats_to_tf_stats_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_kernel_stats_db_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_memory_profile_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_op_metrics_db_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_op_stats_test PASSED in 0.3s //tensorflow/core/profiler/convert:xplane_to_step_events_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_tf_functions_test PASSED in 0.7s //tensorflow/core/profiler/convert:xplane_to_tool_names_test PASSED in 0.2s //tensorflow/core/profiler/internal:tfprof_show_test PASSED in 0.9s //tensorflow/core/profiler/internal:tfprof_stats_test PASSED in 0.9s //tensorflow/core/profiler/internal:tfprof_tensor_test PASSED in 1.5s //tensorflow/core/profiler/internal:tfprof_timeline_test PASSED in 2.0s //tensorflow/core/profiler/internal/advisor:tfprof_advisor_test PASSED in 1.4s //tensorflow/core/profiler/lib:profiler_disabled_test PASSED in 0.1s //tensorflow/core/profiler/utils:derived_timeline_test PASSED in 0.1s //tensorflow/core/profiler/utils:kernel_stats_utils_test PASSED in 0.2s //tensorflow/core/profiler/utils:op_metrics_db_utils_test PASSED in 0.1s //tensorflow/core/profiler/utils:step_intersection_test PASSED in 0.6s //tensorflow/core/summary:schema_test PASSED in 0.1s //tensorflow/core/summary:summary_db_writer_test PASSED in 0.2s //tensorflow/core/summary:summary_file_writer_test PASSED in 0.1s //tensorflow/core/tfrt/common:pjrt_state_test PASSED in 5.4s //tensorflow/core/tfrt/common:pjrt_util_test PASSED in 5.1s //tensorflow/core/tfrt/fallback:cost_recorder_test PASSED in 0.2s //tensorflow/core/tfrt/fallback:fallback_state_test PASSED in 0.6s //tensorflow/core/transforms:eval_utils_test PASSED in 1.6s //tensorflow/core/transforms:graph_transform_wrapper_test PASSED in 0.3s //tensorflow/core/util:bcast_test PASSED in 0.8s //tensorflow/core/util:command_line_flags_test PASSED in 0.7s //tensorflow/core/util:debug_data_dumper_test PASSED in 0.9s //tensorflow/core/util:debug_events_writer_test PASSED in 0.4s //tensorflow/core/util:dump_graph_test PASSED in 1.1s //tensorflow/core/util:equal_graph_def_test PASSED in 0.8s //tensorflow/core/util:events_writer_test PASSED in 2.9s //tensorflow/core/util:example_proto_fast_parsing_test PASSED in 1.7s //tensorflow/core/util:example_proto_helper_test PASSED in 1.3s //tensorflow/core/util:exec_on_stall_test PASSED in 2.3s //tensorflow/core/util:fake_clock_env_test PASSED in 2.1s //tensorflow/core/util:incremental_barrier_test PASSED in 0.1s //tensorflow/core/util:matmul_bcast_test PASSED in 1.5s //tensorflow/core/util:memmapped_file_system_test PASSED in 0.7s //tensorflow/core/util:overflow_test PASSED in 0.1s //tensorflow/core/util:presized_cuckoo_map_test PASSED in 2.2s //tensorflow/core/util:ragged_to_dense_util_test PASSED in 0.9s //tensorflow/core/util:reffed_status_callback_test PASSED in 1.2s //tensorflow/core/util:reporter_test PASSED in 0.8s //tensorflow/core/util:saved_tensor_slice_util_test PASSED in 0.7s //tensorflow/core/util:semver_test PASSED in 0.7s //tensorflow/core/util:stat_summarizer_test PASSED in 0.8s //tensorflow/core/util:strided_slice_op_test PASSED in 0.8s //tensorflow/core/util:tensor_format_test PASSED in 0.6s //tensorflow/core/util:tensor_slice_reader_test PASSED in 2.3s //tensorflow/core/util:tensor_slice_set_test PASSED in 1.2s //tensorflow/core/util:tensor_slice_util_test PASSED in 1.1s //tensorflow/core/util:tensor_slice_writer_test PASSED in 1.6s //tensorflow/core/util:work_sharder_test PASSED in 1.4s //tensorflow/core/util/ctc:ctc_beam_search_test PASSED in 0.3s //tensorflow/core/util/proto:descriptor_pool_registry_test PASSED in 0.7s //tensorflow/core/util/proto:proto_utils_test PASSED in 0.5s //tensorflow/core/util/quantization:uniform_quant_ops_params_test PASSED in 0.8s //tensorflow/core/util/sparse:sparse_tensor_test PASSED in 0.2s //tensorflow/core/util/tensor_bundle:tensor_bundle_test PASSED in 36.9s //tensorflow/dtensor/mlir:dtensor_location_test PASSED in 0.1s //tensorflow/dtensor/mlir:group_assignment_test PASSED in 0.4s //tensorflow/dtensor/mlir/tests:annotate_global_shape.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:cluster_function_conversion.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:constant_folding.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:designate_resource_handle_mesh.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:device_mesh_cluster_coarsening.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:dtensor_all_gather.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_all_scatter.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_combine_optimization.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_lowering.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_scatter_optimization.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_sum_optimization.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:dtensor_layout_must_execute.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:dtensor_mixed_precision_reduce.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_reduce_scatter_lowering.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_remove_dtensorlayout.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_replace_auxiliary_layout_op.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_replace_relayout_with_identity.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding_default.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:dtensor_xla_spmd_integration.mlir.test PASSED in 1.6s //tensorflow/dtensor/mlir/tests:elide_identity_before_copy_to_mesh.mlir.test PASSED in 1.4s //tensorflow/dtensor/mlir/tests:function_renaming.mlir.test PASSED in 0.4s //tensorflow/dtensor/mlir/tests:handle_cross_cluster_dependencies.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:handle_sparsetensors.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:layout_propagation_v2.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:lower_send_recv.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:merge_clusters.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:mesh_propagation.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:op_to_device_cluster.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:propagate_default_layout.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:propagate_device_id_to_function.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:restore_and_assign.mlir.test PASSED in 1.4s //tensorflow/dtensor/mlir/tests:restore_shape_inference.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:set_default_sharding.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:sparse_expansion.mlir.test PASSED in 0.4s //tensorflow/dtensor/mlir/tests:spmd_batchparallel.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:spmd_concat.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_conv.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:spmd_einsum.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:spmd_expansion.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_io_ops.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:spmd_iterator.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_matmul.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_random.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_save_restore.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:spmd_segment_sum.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_slice.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:spmd_softmax_loss.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:spmd_squeeze.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:spmd_var_handle.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:tf_dtensor_ops.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:tpu_add_resource_device_attribute.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:tpu_integration.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:undo_merge_const_across_mesh.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:update_tpu_metadata.mlir.test PASSED in 0.6s //tensorflow/dtensor/python/tests:collective_combine_all_reduce_test_cpu PASSED in 17.4s //tensorflow/dtensor/python/tests:collective_test_cpu PASSED in 16.5s //tensorflow/dtensor/python/tests:config_test_cpu PASSED in 10.7s //tensorflow/dtensor/python/tests:multi_client_test_cpu PASSED in 16.7s //tensorflow/dtensor/python/tests:numpy_util_test_cpu PASSED in 10.3s //tensorflow/dtensor/tests:executable_manager_test PASSED in 29.7s //tensorflow/dtensor/tests:layout_to_xla_sharding_test PASSED in 0.2s //tensorflow/dtensor/tests:tensor_layout_test PASSED in 0.2s //tensorflow/examples/adding_an_op:fact_test PASSED in 16.6s //tensorflow/examples/adding_an_op:zero_out_1_test PASSED in 16.4s //tensorflow/examples/adding_an_op:zero_out_2_test PASSED in 20.2s //tensorflow/examples/adding_an_op:zero_out_3_test PASSED in 14.8s //tensorflow/examples/custom_ops_doc/multiplex_1:multiplex_1_test PASSED in 15.7s //tensorflow/examples/custom_ops_doc/multiplex_2:multiplex_2_test_cpu PASSED in 16.3s //tensorflow/examples/custom_ops_doc/multiplex_3:multiplex_3_test PASSED in 18.4s //tensorflow/examples/custom_ops_doc/multiplex_4:multiplex_4_test PASSED in 28.4s //tensorflow/examples/custom_ops_doc/simple_hash_table:simple_hash_table_test PASSED in 19.0s //tensorflow/examples/custom_ops_doc/sleep:sleep_test PASSED in 15.9s //tensorflow/examples/speech_commands:accuracy_utils_test PASSED in 2.5s //tensorflow/examples/speech_commands:models_test PASSED in 19.1s //tensorflow/examples/speech_commands:recognize_commands_test PASSED in 2.0s //tensorflow/examples/wav_to_spectrogram:wav_to_spectrogram_test PASSED in 2.2s //tensorflow/js:ts_op_gen_test PASSED in 0.6s //tensorflow/python:array_grad_test_cpu PASSED in 10.5s //tensorflow/python:autograph_ops_test PASSED in 7.5s //tensorflow/python:batch_norm_benchmark_cpu PASSED in 8.0s //tensorflow/python:bincount_ops_test PASSED in 10.9s //tensorflow/python:bitwise_ops_test_cpu PASSED in 9.3s //tensorflow/python:clip_ops_test PASSED in 10.2s //tensorflow/python:clustering_ops_test PASSED in 21.2s //tensorflow/python:collective_ops_benchmark_cpu PASSED in 9.3s //tensorflow/python:collective_ops_gpu_test_2gpu PASSED in 10.8s //tensorflow/python:collective_ops_gpu_test_cpu PASSED in 9.6s //tensorflow/python:collective_ops_test PASSED in 32.9s //tensorflow/python:collective_ops_xla_test PASSED in 10.2s //tensorflow/python:compiled_collective_ops_gpu_test_2gpu PASSED in 18.9s //tensorflow/python:compiled_collective_ops_gpu_test_cpu PASSED in 21.7s //tensorflow/python:concat_benchmark_cpu PASSED in 9.3s //tensorflow/python:control_flow_ops_benchmark_cpu PASSED in 8.9s //tensorflow/python:control_flow_v2_enable_test PASSED in 9.7s //tensorflow/python:control_flow_v2_toggles_test PASSED in 7.9s //tensorflow/python:dequantize_op_test PASSED in 15.1s //tensorflow/python:embedding_ops_test_cpu PASSED in 9.6s //tensorflow/python:factory_ops_test_cpu PASSED in 10.9s //tensorflow/python:functional_ops_test PASSED in 8.8s //tensorflow/python:gradient_checker_v2_test_cpu PASSED in 24.8s //tensorflow/python:gradients_test_cpu PASSED in 30.9s //tensorflow/python:init_ops_test_cpu PASSED in 9.8s //tensorflow/python:init_ops_v2_test_cpu PASSED in 12.8s //tensorflow/python:math_grad_test_cpu PASSED in 30.5s //tensorflow/python:math_ops_linspace_test_cpu PASSED in 19.6s //tensorflow/python:math_ops_test_cpu PASSED in 33.9s //tensorflow/python:matmul_benchmark_cpu PASSED in 7.9s //tensorflow/python:nn_grad_test_cpu PASSED in 21.3s //tensorflow/python:nn_loss_scaling_utilities_test PASSED in 10.9s //tensorflow/python:nn_test_cpu PASSED in 85.5s //tensorflow/python:nn_xent_test_cpu PASSED in 9.4s //tensorflow/python:op_selector_test PASSED in 7.5s //tensorflow/python:ops/array_ops_test PASSED in 8.8s //tensorflow/python:quantized_conv_ops_test PASSED in 10.9s //tensorflow/python:quantized_ops_test PASSED in 8.2s //tensorflow/python:raw_ops_test_cpu PASSED in 8.8s //tensorflow/python:rnn_grad_test_cpu PASSED in 10.7s //tensorflow/python:script_ops_test PASSED in 11.0s //tensorflow/python:sort_ops_test PASSED in 9.5s //tensorflow/python:sparse_ops_test PASSED in 16.2s //tensorflow/python:split_benchmark_cpu PASSED in 7.5s //tensorflow/python:tensor_array_ops_test PASSED in 7.1s //tensorflow/python:transpose_benchmark_cpu PASSED in 9.4s //tensorflow/python:variable_spec_test PASSED in 10.2s //tensorflow/python/autograph/converters:asserts_test PASSED in 8.1s //tensorflow/python/autograph/converters:break_statements_test PASSED in 8.6s //tensorflow/python/autograph/converters:call_trees_test PASSED in 8.4s //tensorflow/python/autograph/converters:conditional_expressions_test PASSED in 8.1s //tensorflow/python/autograph/converters:continue_statements_test PASSED in 18.2s //tensorflow/python/autograph/converters:control_flow_test PASSED in 17.8s //tensorflow/python/autograph/converters:directives_test PASSED in 7.1s //tensorflow/python/autograph/converters:functions_test PASSED in 7.6s //tensorflow/python/autograph/converters:list_comprehensions_test PASSED in 8.8s //tensorflow/python/autograph/converters:lists_test PASSED in 7.4s //tensorflow/python/autograph/converters:logical_expressions_test PASSED in 7.3s //tensorflow/python/autograph/converters:return_statements_test PASSED in 21.0s //tensorflow/python/autograph/converters:slices_test PASSED in 11.1s //tensorflow/python/autograph/converters:variables_test PASSED in 9.0s //tensorflow/python/autograph/core:converter_test PASSED in 9.1s //tensorflow/python/autograph/core:function_wrappers_test PASSED in 8.3s //tensorflow/python/autograph/impl:api_test PASSED in 15.2s //tensorflow/python/autograph/impl:conversion_test PASSED in 10.2s //tensorflow/python/autograph/lang:special_functions_test PASSED in 8.3s //tensorflow/python/autograph/operators:conditional_expressions_test PASSED in 9.6s //tensorflow/python/autograph/operators:control_flow_test PASSED in 28.8s //tensorflow/python/autograph/operators:data_structures_test PASSED in 6.9s //tensorflow/python/autograph/operators:exceptions_test PASSED in 10.5s //tensorflow/python/autograph/operators:logical_test PASSED in 10.4s //tensorflow/python/autograph/operators:py_builtins_test PASSED in 30.3s //tensorflow/python/autograph/operators:slices_test PASSED in 18.7s //tensorflow/python/autograph/operators:variables_test PASSED in 18.9s //tensorflow/python/autograph/pyct:anno_test PASSED in 19.2s //tensorflow/python/autograph/pyct:ast_util_test PASSED in 6.8s //tensorflow/python/autograph/pyct:cache_test PASSED in 8.1s //tensorflow/python/autograph/pyct:cfg_test PASSED in 8.0s //tensorflow/python/autograph/pyct:error_utils_test PASSED in 7.1s //tensorflow/python/autograph/pyct:inspect_utils_test PASSED in 8.7s //tensorflow/python/autograph/pyct:loader_test PASSED in 8.5s //tensorflow/python/autograph/pyct:naming_test PASSED in 7.8s //tensorflow/python/autograph/pyct:origin_info_test PASSED in 6.4s //tensorflow/python/autograph/pyct:parser_test PASSED in 7.5s //tensorflow/python/autograph/pyct:pretty_printer_test PASSED in 14.6s //tensorflow/python/autograph/pyct:qual_names_test PASSED in 8.3s //tensorflow/python/autograph/pyct:templates_test PASSED in 9.6s //tensorflow/python/autograph/pyct:transformer_test PASSED in 6.7s //tensorflow/python/autograph/pyct:transpiler_test PASSED in 14.5s //tensorflow/python/autograph/pyct/static_analysis:activity_test PASSED in 6.5s //tensorflow/python/autograph/pyct/static_analysis:liveness_test PASSED in 7.9s //tensorflow/python/autograph/pyct/static_analysis:reaching_definitions_test PASSED in 12.7s //tensorflow/python/autograph/pyct/static_analysis:type_inference_test PASSED in 8.8s //tensorflow/python/autograph/tests:assertion_test PASSED in 29.0s //tensorflow/python/autograph/tests:basic_ifexp_test PASSED in 20.0s //tensorflow/python/autograph/tests:call_to_builtin_function_test PASSED in 35.8s //tensorflow/python/autograph/tests:call_to_lambda_function_test PASSED in 18.8s //tensorflow/python/autograph/tests:call_to_named_tuple_test PASSED in 16.2s //tensorflow/python/autograph/tests:call_to_numpy_function_test PASSED in 18.2s //tensorflow/python/autograph/tests:call_to_print_function_test PASSED in 36.9s //tensorflow/python/autograph/tests:call_to_tf_api_test PASSED in 16.5s //tensorflow/python/autograph/tests:call_to_user_function_test PASSED in 20.5s //tensorflow/python/autograph/tests:composite_names_in_control_flow_test PASSED in 47.6s //tensorflow/python/autograph/tests:cond_basic_test PASSED in 46.1s //tensorflow/python/autograph/tests:datasets_test PASSED in 32.5s //tensorflow/python/autograph/tests:early_return_test PASSED in 25.8s //tensorflow/python/autograph/tests:ext_slice_test PASSED in 38.6s //tensorflow/python/autograph/tests:generator_test PASSED in 19.5s //tensorflow/python/autograph/tests:logical_expression_test PASSED in 17.3s //tensorflow/python/autograph/tests:loop_basic_test PASSED in 91.7s //tensorflow/python/autograph/tests:loop_control_flow_illegal_cases_test PASSED in 28.5s //tensorflow/python/autograph/tests:loop_created_variables_test PASSED in 23.9s //tensorflow/python/autograph/tests:loop_scoping_test PASSED in 19.0s //tensorflow/python/autograph/tests:loop_with_function_call_test PASSED in 38.1s //tensorflow/python/autograph/tests:loop_with_variable_type_illegal_cases_test PASSED in 23.8s //tensorflow/python/autograph/tests:loop_with_variable_type_test PASSED in 35.3s //tensorflow/python/autograph/tests:nested_control_flow_test PASSED in 57.1s //tensorflow/python/autograph/tests:type_annotations_test PASSED in 12.6s //tensorflow/python/autograph/utils:context_managers_test PASSED in 7.7s //tensorflow/python/autograph/utils:misc_test PASSED in 9.7s //tensorflow/python/autograph/utils:tensor_list_test PASSED in 7.8s //tensorflow/python/autograph/utils:tensors_test PASSED in 8.0s //tensorflow/python/checkpoint:benchmarks_test PASSED in 7.3s //tensorflow/python/checkpoint:checkpoint_management_test_cpu PASSED in 34.3s //tensorflow/python/checkpoint:checkpoint_metrics_test PASSED in 36.6s //tensorflow/python/checkpoint:checkpoint_test PASSED in 30.5s //tensorflow/python/checkpoint:checkpoint_view_test PASSED in 21.8s //tensorflow/python/checkpoint:checkpoint_with_v1_optimizers_test PASSED in 33.8s //tensorflow/python/checkpoint:functional_saver_test_cpu PASSED in 12.2s //tensorflow/python/checkpoint:restore_test PASSED in 25.2s //tensorflow/python/checkpoint:save_util_v1_test PASSED in 10.9s //tensorflow/python/checkpoint:saveable_compat_test PASSED in 9.8s //tensorflow/python/checkpoint:tensor_callable_test PASSED in 12.0s //tensorflow/python/checkpoint:trackable_view_test PASSED in 31.2s //tensorflow/python/client:device_lib_test_cpu PASSED in 8.3s //tensorflow/python/client:events_writer_test PASSED in 11.0s //tensorflow/python/client:session_benchmark_cpu PASSED in 11.1s //tensorflow/python/client:session_list_devices_test PASSED in 9.1s //tensorflow/python/client:session_partial_run_test PASSED in 27.0s //tensorflow/python/client:timeline_test_cpu PASSED in 9.1s //tensorflow/python/client:virtual_gpu_test_cpu PASSED in 10.4s //tensorflow/python/compat:compat_test PASSED in 8.0s //tensorflow/python/compat:disable_v2_behavior_test PASSED in 8.0s //tensorflow/python/compiler/mlir:mlir_test PASSED in 19.7s //tensorflow/python/compiler/tensorrt:trt_convert_test_cpu PASSED in 15.3s //tensorflow/python/compiler/tensorrt/test:batch_matmul_test_cpu PASSED in 8.3s //tensorflow/python/compiler/tensorrt/test:biasadd_matmul_test_cpu PASSED in 7.8s //tensorflow/python/compiler/tensorrt/test:binary_tensor_weight_broadcast_test_cpu PASSED in 17.0s //tensorflow/python/compiler/tensorrt/test:bool_test_cpu PASSED in 28.7s //tensorflow/python/compiler/tensorrt/test:cast_test_cpu PASSED in 9.6s //tensorflow/python/compiler/tensorrt/test:concatenation_test_cpu PASSED in 9.6s //tensorflow/python/compiler/tensorrt/test:const_broadcast_test_cpu PASSED in 10.3s //tensorflow/python/compiler/tensorrt/test:data_dependent_shape_test_cpu PASSED in 10.8s //tensorflow/python/compiler/tensorrt/test:dynamic_input_shapes_test_cpu PASSED in 9.7s //tensorflow/python/compiler/tensorrt/test:identity_output_test_cpu PASSED in 30.2s //tensorflow/python/compiler/tensorrt/test:int32_test_cpu PASSED in 8.6s //tensorflow/python/compiler/tensorrt/test:lru_cache_test_cpu PASSED in 9.1s //tensorflow/python/compiler/tensorrt/test:memory_alignment_test_cpu PASSED in 10.6s //tensorflow/python/compiler/tensorrt/test:multi_connection_neighbor_engine_test_cpu PASSED in 8.9s //tensorflow/python/compiler/tensorrt/test:neighboring_engine_test_cpu PASSED in 8.9s //tensorflow/python/compiler/tensorrt/test:quantization_test_cpu PASSED in 10.2s //tensorflow/python/compiler/tensorrt/test:rank_two_test_cpu PASSED in 10.5s //tensorflow/python/compiler/tensorrt/test:reshape_transpose_test_cpu PASSED in 11.7s //tensorflow/python/compiler/tensorrt/test:topk_test_cpu PASSED in 24.3s //tensorflow/python/compiler/tensorrt/test:trt_engine_op_shape_test_cpu PASSED in 11.4s //tensorflow/python/compiler/tensorrt/test:trt_mode_test_cpu PASSED in 10.3s //tensorflow/python/compiler/tensorrt/test:unary_test_cpu PASSED in 14.0s //tensorflow/python/compiler/tensorrt/test:vgg_block_nchw_test_cpu PASSED in 8.8s //tensorflow/python/compiler/tensorrt/test:vgg_block_test_cpu PASSED in 9.0s //tensorflow/python/compiler/xla:jit_compile_test_cpu PASSED in 10.0s //tensorflow/python/compiler/xla:jit_test_cpu PASSED in 29.5s //tensorflow/python/compiler/xla:xla_test_cpu PASSED in 16.9s //tensorflow/python/compiler/xla/experimental:xla_sharding_test PASSED in 27.2s //tensorflow/python/data/benchmarks:batch_benchmark PASSED in 12.4s //tensorflow/python/data/benchmarks:filter_benchmark PASSED in 9.6s //tensorflow/python/data/benchmarks:from_tensor_slices_benchmark PASSED in 12.5s //tensorflow/python/data/benchmarks:interleave_benchmark PASSED in 8.7s //tensorflow/python/data/benchmarks:list_files_benchmark PASSED in 9.7s //tensorflow/python/data/benchmarks:map_benchmark PASSED in 9.6s //tensorflow/python/data/benchmarks:meta_benchmark PASSED in 10.5s //tensorflow/python/data/benchmarks:prefetch_benchmark PASSED in 9.4s //tensorflow/python/data/benchmarks:range_benchmark PASSED in 9.1s //tensorflow/python/data/experimental/benchmarks:autotune_benchmark PASSED in 9.5s //tensorflow/python/data/experimental/benchmarks:csv_dataset_benchmark PASSED in 8.1s //tensorflow/python/data/experimental/benchmarks:map_and_batch_benchmark PASSED in 9.6s //tensorflow/python/data/experimental/benchmarks:map_defun_benchmark PASSED in 10.0s //tensorflow/python/data/experimental/benchmarks:matching_files_benchmark PASSED in 8.7s //tensorflow/python/data/experimental/benchmarks:optimize_benchmark PASSED in 6.8s //tensorflow/python/data/experimental/benchmarks:parameter_value_benchmark PASSED in 9.9s //tensorflow/python/data/experimental/benchmarks:rejection_resample_benchmark PASSED in 10.3s //tensorflow/python/data/experimental/benchmarks:snapshot_dataset_benchmark PASSED in 10.7s //tensorflow/python/data/experimental/benchmarks:unbatch_benchmark PASSED in 10.6s //tensorflow/python/data/experimental/kernel_tests:assert_cardinality_test PASSED in 26.4s //tensorflow/python/data/experimental/kernel_tests:assert_next_test PASSED in 14.6s //tensorflow/python/data/experimental/kernel_tests:assert_prev_test PASSED in 11.3s //tensorflow/python/data/experimental/kernel_tests:checkpoint_input_pipeline_hook_test PASSED in 13.8s //tensorflow/python/data/experimental/kernel_tests:compression_ops_test PASSED in 13.6s //tensorflow/python/data/experimental/kernel_tests:copy_to_device_test_cpu PASSED in 20.3s //tensorflow/python/data/experimental/kernel_tests:dense_to_sparse_batch_test PASSED in 18.0s //tensorflow/python/data/experimental/kernel_tests:from_list_test PASSED in 28.8s //tensorflow/python/data/experimental/kernel_tests:io_test PASSED in 56.2s //tensorflow/python/data/experimental/kernel_tests:lookup_ops_test PASSED in 11.2s //tensorflow/python/data/experimental/kernel_tests:make_csv_dataset_test PASSED in 21.4s //tensorflow/python/data/experimental/kernel_tests:make_saveable_from_iterator_test PASSED in 7.7s //tensorflow/python/data/experimental/kernel_tests:make_tf_record_dataset_test PASSED in 94.4s //tensorflow/python/data/experimental/kernel_tests:map_defun_op_test PASSED in 9.9s //tensorflow/python/data/experimental/kernel_tests:matching_files_dataset_test PASSED in 31.7s //tensorflow/python/data/experimental/kernel_tests:model_dataset_test PASSED in 8.4s //tensorflow/python/data/experimental/kernel_tests:non_serializable_test PASSED in 10.7s //tensorflow/python/data/experimental/kernel_tests:prefetch_to_device_test_cpu PASSED in 11.5s //tensorflow/python/data/experimental/kernel_tests:prefetch_with_slack_test PASSED in 9.7s //tensorflow/python/data/experimental/kernel_tests:shuffle_and_repeat_test PASSED in 17.6s //tensorflow/python/data/experimental/kernel_tests:sleep_test PASSED in 8.6s //tensorflow/python/data/experimental/kernel_tests:tf_record_writer_test PASSED in 11.4s //tensorflow/python/data/experimental/kernel_tests:variant_test PASSED in 8.7s //tensorflow/python/data/experimental/kernel_tests:wrap_unwrap_test_cpu PASSED in 8.6s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_fusion_test PASSED in 28.6s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_parallelization_test PASSED in 52.4s //tensorflow/python/data/experimental/kernel_tests/optimization:grappler_test_cpu PASSED in 8.5s //tensorflow/python/data/experimental/kernel_tests/optimization:make_deterministic_test PASSED in 22.4s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_batch_fusion_test PASSED in 8.4s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_filter_fusion_test PASSED in 28.1s //tensorflow/python/data/experimental/kernel_tests/optimization:map_fusion_test PASSED in 19.6s //tensorflow/python/data/experimental/kernel_tests/optimization:map_parallelization_test PASSED in 11.2s //tensorflow/python/data/experimental/kernel_tests/optimization:noop_elimination_test PASSED in 11.1s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_test PASSED in 10.6s //tensorflow/python/data/experimental/kernel_tests/service:multi_device_test PASSED in 24.9s //tensorflow/python/data/experimental/service:server_lib_test PASSED in 9.8s //tensorflow/python/data/kernel_tests:as_numpy_iterator_test PASSED in 18.6s //tensorflow/python/data/kernel_tests:bucket_by_sequence_length_test PASSED in 19.6s //tensorflow/python/data/kernel_tests:cache_test PASSED in 45.0s //tensorflow/python/data/kernel_tests:cardinality_test PASSED in 12.2s //tensorflow/python/data/kernel_tests:checkpoint_test PASSED in 18.6s //tensorflow/python/data/kernel_tests:concatenate_test PASSED in 24.7s //tensorflow/python/data/kernel_tests:counter_test PASSED in 28.2s //tensorflow/python/data/kernel_tests:dataset_spec_test PASSED in 9.8s //tensorflow/python/data/kernel_tests:dataset_test PASSED in 26.6s //tensorflow/python/data/kernel_tests:enumerate_test PASSED in 24.7s //tensorflow/python/data/kernel_tests:from_sparse_tensor_slices_test PASSED in 10.1s //tensorflow/python/data/kernel_tests:from_tensor_slices_test PASSED in 58.3s //tensorflow/python/data/kernel_tests:from_tensors_test PASSED in 26.5s //tensorflow/python/data/kernel_tests:get_single_element_test PASSED in 10.6s //tensorflow/python/data/kernel_tests:ignore_errors_test PASSED in 18.4s //tensorflow/python/data/kernel_tests:io_test PASSED in 19.7s //tensorflow/python/data/kernel_tests:iterator_test_cpu PASSED in 31.8s //tensorflow/python/data/kernel_tests:len_test PASSED in 6.6s //tensorflow/python/data/kernel_tests:list_files_test PASSED in 10.0s //tensorflow/python/data/kernel_tests:optional_test_cpu PASSED in 13.2s //tensorflow/python/data/kernel_tests:options_test PASSED in 12.5s //tensorflow/python/data/kernel_tests:placement_test_cpu PASSED in 9.3s //tensorflow/python/data/kernel_tests:prefetch_test PASSED in 51.0s //tensorflow/python/data/kernel_tests:random_test PASSED in 23.0s //tensorflow/python/data/kernel_tests:range_test PASSED in 33.9s //tensorflow/python/data/kernel_tests:rebatch_test PASSED in 11.6s //tensorflow/python/data/kernel_tests:reduce_test_cpu PASSED in 20.0s //tensorflow/python/data/kernel_tests:scan_test_cpu PASSED in 39.5s //tensorflow/python/data/kernel_tests:sparse_batch_test PASSED in 18.5s //tensorflow/python/data/kernel_tests:unbatch_test PASSED in 44.4s //tensorflow/python/data/util:convert_test PASSED in 9.6s //tensorflow/python/data/util:nest_test PASSED in 17.1s //tensorflow/python/data/util:options_test PASSED in 6.9s //tensorflow/python/data/util:random_seed_test PASSED in 11.2s //tensorflow/python/data/util:sparse_test PASSED in 7.9s //tensorflow/python/data/util:structure_test PASSED in 12.8s //tensorflow/python/data/util:traverse_test PASSED in 30.8s //tensorflow/python/debug/cli:analyzer_cli_test_cpu PASSED in 10.7s //tensorflow/python/debug/cli:cli_config_test PASSED in 7.7s //tensorflow/python/debug/cli:cli_shared_test PASSED in 6.4s //tensorflow/python/debug/cli:command_parser_test PASSED in 8.2s //tensorflow/python/debug/cli:curses_ui_test PASSED in 6.6s //tensorflow/python/debug/cli:debugger_cli_common_test PASSED in 6.5s //tensorflow/python/debug/cli:evaluator_test PASSED in 7.0s //tensorflow/python/debug/cli:profile_analyzer_cli_test PASSED in 6.8s //tensorflow/python/debug/cli:readline_ui_test PASSED in 7.1s //tensorflow/python/debug/cli:tensor_format_test PASSED in 6.4s //tensorflow/python/debug/lib:check_numerics_callback_test_cpu PASSED in 13.1s //tensorflow/python/debug/lib:common_test PASSED in 5.6s //tensorflow/python/debug/lib:debug_data_test PASSED in 6.6s //tensorflow/python/debug/lib:debug_events_monitors_test PASSED in 9.0s //tensorflow/python/debug/lib:debug_events_writer_test PASSED in 9.6s //tensorflow/python/debug/lib:debug_gradients_test_cpu PASSED in 8.4s //tensorflow/python/debug/lib:debug_graph_reconstruction_test_cpu PASSED in 10.3s //tensorflow/python/debug/lib:debug_graphs_test PASSED in 6.0s //tensorflow/python/debug/lib:debug_grappler_test_cpu PASSED in 14.2s //tensorflow/python/debug/lib:debug_utils_test PASSED in 7.1s //tensorflow/python/debug/lib:debug_v2_ops_test_cpu PASSED in 16.8s //tensorflow/python/debug/lib:profiling_test PASSED in 6.4s //tensorflow/python/debug/lib:session_debug_file_test_cpu PASSED in 13.0s //tensorflow/python/debug/lib:session_debug_multi_gpu_test_cpu PASSED in 7.3s //tensorflow/python/debug/lib:source_utils_test PASSED in 9.8s //tensorflow/python/debug/wrappers:disk_usage_test PASSED in 9.1s //tensorflow/python/debug/wrappers:dumping_wrapper_test PASSED in 9.8s //tensorflow/python/debug/wrappers:framework_test PASSED in 9.9s //tensorflow/python/debug/wrappers:local_cli_wrapper_test PASSED in 6.8s //tensorflow/python/distribute:checkpoint_utils_test_2gpu PASSED in 13.9s //tensorflow/python/distribute:checkpoint_utils_test_cpu PASSED in 12.5s //tensorflow/python/distribute:checkpointing_test_2gpu PASSED in 11.0s //tensorflow/python/distribute:checkpointing_test_cpu PASSED in 9.0s //tensorflow/python/distribute:collective_all_reduce_strategy_test_2gpu PASSED in 74.4s //tensorflow/python/distribute:collective_all_reduce_strategy_test_cpu PASSED in 70.1s //tensorflow/python/distribute:collective_all_reduce_strategy_test_xla_2gpu PASSED in 23.6s //tensorflow/python/distribute:collective_util_test PASSED in 8.2s //tensorflow/python/distribute:combinations_test_2gpu PASSED in 20.5s //tensorflow/python/distribute:combinations_test_cpu PASSED in 19.6s //tensorflow/python/distribute:cross_device_utils_test_cpu PASSED in 21.4s //tensorflow/python/distribute:custom_training_loop_gradient_test_2gpu PASSED in 10.1s //tensorflow/python/distribute:custom_training_loop_gradient_test_cpu PASSED in 10.2s //tensorflow/python/distribute:device_util_test_cpu PASSED in 10.6s //tensorflow/python/distribute:distribute_coordinator_test PASSED in 16.6s //tensorflow/python/distribute:distribute_lib_test PASSED in 13.9s //tensorflow/python/distribute:distribute_utils_test_2gpu PASSED in 9.5s //tensorflow/python/distribute:distribute_utils_test_cpu PASSED in 10.0s //tensorflow/python/distribute:input_ops_test_cpu PASSED in 13.4s //tensorflow/python/distribute:metrics_v1_test_2gpu PASSED in 36.7s //tensorflow/python/distribute:metrics_v1_test_cpu PASSED in 31.1s //tensorflow/python/distribute:mirrored_values_test_2gpu PASSED in 12.7s //tensorflow/python/distribute:mirrored_values_test_cpu PASSED in 20.0s //tensorflow/python/distribute:mirrored_variable_test_2gpu PASSED in 24.6s //tensorflow/python/distribute:mirrored_variable_test_cpu PASSED in 34.0s //tensorflow/python/distribute:multi_process_runner_no_init_test PASSED in 19.9s //tensorflow/python/distribute:multi_worker_continuous_run_test_cpu PASSED in 19.7s //tensorflow/python/distribute:multi_worker_util_test PASSED in 8.2s //tensorflow/python/distribute:numpy_dataset_test PASSED in 7.5s //tensorflow/python/distribute:one_device_strategy_test_cpu PASSED in 38.1s //tensorflow/python/distribute:packed_distributed_variable_test PASSED in 9.3s //tensorflow/python/distribute:parameter_server_strategy_test_2gpu PASSED in 27.3s //tensorflow/python/distribute:parameter_server_strategy_test_cpu PASSED in 30.9s //tensorflow/python/distribute:parameter_server_strategy_v2_test_2gpu PASSED in 22.9s //tensorflow/python/distribute:parameter_server_strategy_v2_test_cpu PASSED in 21.7s //tensorflow/python/distribute:per_replica_test_2gpu PASSED in 10.0s //tensorflow/python/distribute:per_replica_test_cpu PASSED in 9.4s //tensorflow/python/distribute:ps_values_test_2gpu PASSED in 8.8s //tensorflow/python/distribute:ps_values_test_cpu PASSED in 11.1s //tensorflow/python/distribute:remote_mirrored_strategy_eager_test_cpu PASSED in 9.7s //tensorflow/python/distribute:sharded_variable_test PASSED in 20.7s //tensorflow/python/distribute:shared_variable_creator_test PASSED in 6.4s //tensorflow/python/distribute:strategy_combinations_test_cpu PASSED in 47.7s //tensorflow/python/distribute:template_mirrored_strategy_test_cpu PASSED in 8.4s //tensorflow/python/distribute:test_util_test_2gpu PASSED in 14.9s //tensorflow/python/distribute:test_util_test_cpu PASSED in 15.8s //tensorflow/python/distribute:tf_function_test_2gpu PASSED in 10.8s //tensorflow/python/distribute:tf_function_test_cpu PASSED in 9.9s //tensorflow/python/distribute:values_v2_test_cpu PASSED in 11.5s //tensorflow/python/distribute:warm_starting_util_test_2gpu PASSED in 11.2s //tensorflow/python/distribute:warm_starting_util_test_cpu PASSED in 10.9s //tensorflow/python/distribute/cluster_resolver:base_cluster_resolver_py_test PASSED in 7.7s //tensorflow/python/distribute/cluster_resolver:gce_cluster_resolver_py_test PASSED in 7.5s //tensorflow/python/distribute/cluster_resolver:kubernetes_cluster_resolver_py_test PASSED in 10.0s //tensorflow/python/distribute/cluster_resolver:sagemaker_cluster_resolver_py_test PASSED in 10.1s //tensorflow/python/distribute/cluster_resolver:slurm_cluster_resolver_py_test PASSED in 8.1s //tensorflow/python/distribute/cluster_resolver:tfconfig_cluster_resolver_py_test PASSED in 9.2s //tensorflow/python/distribute/cluster_resolver/tpu:tpu_cluster_resolver_py_test PASSED in 9.9s //tensorflow/python/distribute/coordinator:metric_utils_test PASSED in 10.7s //tensorflow/python/distribute/coordinator:watchdog_test PASSED in 63.2s //tensorflow/python/distribute/experimental:dtensor_util_test_cpu PASSED in 12.8s //tensorflow/python/distribute/experimental:mirrored_strategy_test_cpu PASSED in 33.3s //tensorflow/python/distribute/integration_test:saved_model_test_cpu PASSED in 51.2s //tensorflow/python/distribute/parallel_device:parallel_device_test_cpu PASSED in 19.2s //tensorflow/python/distribute/v1:all_reduce_test PASSED in 60.1s //tensorflow/python/distribute/v1:cross_device_ops_test_2gpu PASSED in 56.0s //tensorflow/python/distribute/v1:cross_device_ops_test_cpu PASSED in 50.9s //tensorflow/python/dlpack:dlpack_test_cpu PASSED in 8.1s //tensorflow/python/eager:backprop_test_cpu PASSED in 120.1s //tensorflow/python/eager:benchmarks_test_cpu PASSED in 9.4s //tensorflow/python/eager:cancellation_test_cpu PASSED in 8.2s //tensorflow/python/eager:context_test_cpu PASSED in 10.5s //tensorflow/python/eager:core_test_cpu PASSED in 17.6s //tensorflow/python/eager:gradient_input_output_exclusions_test PASSED in 34.2s //tensorflow/python/eager:graph_only_ops_test_cpu PASSED in 10.6s //tensorflow/python/eager:lift_to_graph_test PASSED in 9.8s //tensorflow/python/eager:monitoring_test_cpu PASSED in 8.5s //tensorflow/python/eager:ops_test_cpu PASSED in 14.5s //tensorflow/python/eager:profiler_client_test PASSED in 6.7s //tensorflow/python/eager:profiler_test_cpu PASSED in 9.0s //tensorflow/python/eager:pywrap_tfe_test PASSED in 22.2s //tensorflow/python/eager:remote_benchmarks_test_cpu PASSED in 7.9s //tensorflow/python/eager:run_eager_op_as_function_test_cpu PASSED in 8.7s //tensorflow/python/eager:run_eager_op_as_function_xla_test_cpu PASSED in 9.0s //tensorflow/python/eager:tape_test PASSED in 11.9s //tensorflow/python/eager:tensor_test_cpu PASSED in 14.2s //tensorflow/python/eager:wrap_function_device_test_cpu PASSED in 10.3s //tensorflow/python/eager:wrap_function_test PASSED in 16.1s //tensorflow/python/eager/benchmarks:kpi_benchmark_test_cpu PASSED in 16.7s //tensorflow/python/eager/memory_tests:remote_memory_test_cpu PASSED in 8.8s //tensorflow/python/eager/polymorphic_function:argument_naming_test_cpu PASSED in 9.7s //tensorflow/python/eager/polymorphic_function:collection_test_cpu PASSED in 9.8s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu PASSED in 9.3s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu_mlir_bridge_test PASSED in 9.0s //tensorflow/python/eager/polymorphic_function:function_spec_test PASSED in 9.3s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_jit_test_cpu PASSED in 41.1s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_jit_test_cpu_mlir_bridge_test PASSED in 34.9s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_test_cpu PASSED in 12.5s //tensorflow/python/eager/polymorphic_function:quarantine_test PASSED in 32.7s //tensorflow/python/feature_column:sequence_feature_column_integration_test PASSED in 9.3s //tensorflow/python/feature_column:serialization_test PASSED in 14.5s //tensorflow/python/framework:auto_control_deps_test PASSED in 24.7s //tensorflow/python/framework:c_api_util_test PASSED in 7.4s //tensorflow/python/framework:common_shapes_test PASSED in 7.4s //tensorflow/python/framework:composite_tensor_test PASSED in 12.7s //tensorflow/python/framework:config_test_2gpu PASSED in 12.4s //tensorflow/python/framework:config_test_cpu PASSED in 12.5s //tensorflow/python/framework:constant_op_test PASSED in 9.4s //tensorflow/python/framework:device_spec_test PASSED in 8.2s //tensorflow/python/framework:device_test PASSED in 8.3s //tensorflow/python/framework:dtypes_test PASSED in 17.3s //tensorflow/python/framework:error_interpolation_test PASSED in 8.6s //tensorflow/python/framework:errors_test PASSED in 8.8s //tensorflow/python/framework:extension_type_field_test PASSED in 7.4s //tensorflow/python/framework:extension_type_test PASSED in 21.8s //tensorflow/python/framework:file_system_test PASSED in 17.7s //tensorflow/python/framework:function_def_to_graph_test PASSED in 9.2s //tensorflow/python/framework:graph_building_benchmark_cpu PASSED in 8.8s //tensorflow/python/framework:graph_util_test PASSED in 8.9s //tensorflow/python/framework:immutable_dict_test PASSED in 9.7s //tensorflow/python/framework:importer_test PASSED in 7.5s //tensorflow/python/framework:indexed_slices_test PASSED in 9.7s //tensorflow/python/framework:kernels_test PASSED in 7.4s //tensorflow/python/framework:meta_graph_test PASSED in 9.2s //tensorflow/python/framework:node_file_writer_test_cpu PASSED in 7.5s //tensorflow/python/framework:offset_counter_helper_test PASSED in 0.8s //tensorflow/python/framework:op_allowlist_namespace_test PASSED in 1.9s //tensorflow/python/framework:op_callbacks_test_cpu PASSED in 11.1s //tensorflow/python/framework:op_def_library_test PASSED in 11.2s //tensorflow/python/framework:op_def_util_test PASSED in 7.8s //tensorflow/python/framework:ops_enable_eager_test PASSED in 2.3s //tensorflow/python/framework:ops_test PASSED in 22.0s //tensorflow/python/framework:proto_test PASSED in 11.3s //tensorflow/python/framework:py_context_manager_test PASSED in 9.0s //tensorflow/python/framework:python_api_dispatcher_test PASSED in 8.4s //tensorflow/python/framework:python_api_info_test PASSED in 9.0s //tensorflow/python/framework:python_api_parameter_converter_test PASSED in 11.3s //tensorflow/python/framework:python_op_gen_annotation_test PASSED in 4.5s //tensorflow/python/framework:python_op_gen_annotator_test PASSED in 1.1s //tensorflow/python/framework:python_op_gen_test PASSED in 0.2s //tensorflow/python/framework:python_tensor_converter_test PASSED in 32.9s //tensorflow/python/framework:random_seed_test PASSED in 7.4s //tensorflow/python/framework:registry_test PASSED in 8.9s //tensorflow/python/framework:smart_cond_test PASSED in 8.0s //tensorflow/python/framework:sparse_tensor_test PASSED in 11.0s //tensorflow/python/framework:subscribe_test PASSED in 7.1s //tensorflow/python/framework:tensor_shape_test PASSED in 11.5s //tensorflow/python/framework:tensor_test PASSED in 7.7s //tensorflow/python/framework:tensor_util_test PASSED in 9.6s //tensorflow/python/framework:test_combinations_test PASSED in 7.7s //tensorflow/python/framework:test_util_test_cpu PASSED in 20.7s //tensorflow/python/framework:tf2_test PASSED in 8.7s //tensorflow/python/framework:traceable_stack_test PASSED in 6.7s //tensorflow/python/framework:type_spec_test PASSED in 8.5s //tensorflow/python/framework:versions_test PASSED in 18.6s //tensorflow/python/framework/experimental:graph_building_test_cpu PASSED in 8.1s //tensorflow/python/framework/experimental:unified_api_test_cpu PASSED in 17.4s //tensorflow/python/grappler:arithmetic_optimizer_test_cpu PASSED in 19.8s //tensorflow/python/grappler:auto_mixed_precision_test_cpu PASSED in 11.8s //tensorflow/python/grappler:constant_folding_test_cpu PASSED in 12.6s //tensorflow/python/grappler:cost_analyzer_test PASSED in 8.4s //tensorflow/python/grappler:datasets_test PASSED in 10.3s //tensorflow/python/grappler:item_test PASSED in 8.1s //tensorflow/python/grappler:memory_optimizer_test PASSED in 15.9s //tensorflow/python/grappler:model_analyzer_test PASSED in 10.4s //tensorflow/python/grappler:remapper_test_cpu PASSED in 8.5s //tensorflow/python/grappler:tf_optimizer_test PASSED in 8.8s //tensorflow/python/kernel_tests:benchmark_test_cpu PASSED in 10.7s //tensorflow/python/kernel_tests:check_ops_test_cpu PASSED in 21.4s //tensorflow/python/kernel_tests:collective_ops_multi_worker_test PASSED in 27.0s //tensorflow/python/kernel_tests:composite_tensor_ops_test PASSED in 17.7s //tensorflow/python/kernel_tests:critical_section_test_cpu PASSED in 19.9s //tensorflow/python/kernel_tests:garbage_collection_test PASSED in 9.4s //tensorflow/python/kernel_tests:gradient_correctness_test_cpu PASSED in 9.3s //tensorflow/python/kernel_tests:histogram_ops_test_cpu PASSED in 35.7s //tensorflow/python/kernel_tests:logging_ops_test_cpu PASSED in 13.1s //tensorflow/python/kernel_tests:numerics_test_cpu PASSED in 13.0s //tensorflow/python/kernel_tests:template_test PASSED in 11.4s //tensorflow/python/kernel_tests:trace_op_test_cpu PASSED in 8.3s //tensorflow/python/kernel_tests/array_ops:batch_gather_op_test_cpu PASSED in 9.9s //tensorflow/python/kernel_tests/array_ops:batch_scatter_ops_test PASSED in 11.8s //tensorflow/python/kernel_tests/array_ops:batchtospace_op_test_cpu PASSED in 14.7s //tensorflow/python/kernel_tests/array_ops:bcast_ops_test PASSED in 8.3s //tensorflow/python/kernel_tests/array_ops:bitcast_op_test_cpu PASSED in 12.2s //tensorflow/python/kernel_tests/array_ops:broadcast_to_ops_test_cpu PASSED in 25.2s //tensorflow/python/kernel_tests/array_ops:cast_op_test_cpu PASSED in 9.1s //tensorflow/python/kernel_tests/array_ops:constant_op_eager_test_cpu PASSED in 11.4s //tensorflow/python/kernel_tests/array_ops:constant_op_test_cpu PASSED in 20.7s //tensorflow/python/kernel_tests/array_ops:denormal_test_cpu PASSED in 8.3s //tensorflow/python/kernel_tests/array_ops:depthtospace_op_test_cpu PASSED in 12.7s //tensorflow/python/kernel_tests/array_ops:edit_distance_op_test PASSED in 9.7s //tensorflow/python/kernel_tests/array_ops:fingerprint_op_test PASSED in 7.5s //tensorflow/python/kernel_tests/array_ops:gather_nd_op_test_cpu PASSED in 8.5s //tensorflow/python/kernel_tests/array_ops:identity_n_op_py_test PASSED in 14.3s //tensorflow/python/kernel_tests/array_ops:identity_op_py_test PASSED in 8.5s //tensorflow/python/kernel_tests/array_ops:large_concat_op_test_cpu PASSED in 9.2s //tensorflow/python/kernel_tests/array_ops:manip_ops_test_cpu PASSED in 11.0s //tensorflow/python/kernel_tests/array_ops:one_hot_op_test_cpu PASSED in 16.9s //tensorflow/python/kernel_tests/array_ops:pad_op_test_cpu PASSED in 16.5s //tensorflow/python/kernel_tests/array_ops:reshape_op_test_cpu PASSED in 7.9s //tensorflow/python/kernel_tests/array_ops:reverse_sequence_op_test_cpu PASSED in 9.5s //tensorflow/python/kernel_tests/array_ops:scalar_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/array_ops:shape_ops_test_cpu PASSED in 26.7s //tensorflow/python/kernel_tests/array_ops:slice_op_test_cpu PASSED in 9.2s //tensorflow/python/kernel_tests/array_ops:spacetobatch_op_test_cpu PASSED in 15.2s //tensorflow/python/kernel_tests/array_ops:spacetodepth_op_test_cpu PASSED in 10.7s //tensorflow/python/kernel_tests/array_ops:stack_op_test_cpu PASSED in 14.9s //tensorflow/python/kernel_tests/array_ops:unique_op_test_cpu PASSED in 8.5s //tensorflow/python/kernel_tests/array_ops:unstack_op_test_cpu PASSED in 11.5s //tensorflow/python/kernel_tests/array_ops:where_op_test_cpu PASSED in 18.4s //tensorflow/python/kernel_tests/control_flow:cond_v2_test_cpu PASSED in 54.3s //tensorflow/python/kernel_tests/control_flow:control_flow_util_test PASSED in 9.6s //tensorflow/python/kernel_tests/control_flow:control_flow_util_v2_test PASSED in 10.2s //tensorflow/python/kernel_tests/control_flow:py_func_test_cpu PASSED in 18.7s //tensorflow/python/kernel_tests/control_flow:scan_ops_test_cpu PASSED in 66.8s //tensorflow/python/kernel_tests/control_flow:while_v2_test_cpu PASSED in 58.6s //tensorflow/python/kernel_tests/custom_ops:ackermann_test PASSED in 8.2s //tensorflow/python/kernel_tests/custom_ops:duplicate_op_test PASSED in 7.6s //tensorflow/python/kernel_tests/custom_ops:invalid_op_test PASSED in 8.1s //tensorflow/python/kernel_tests/data_structures:conditional_accumulator_test PASSED in 10.8s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_2gpu PASSED in 13.7s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_cpu PASSED in 14.1s //tensorflow/python/kernel_tests/data_structures:dynamic_stitch_op_test_cpu PASSED in 9.7s //tensorflow/python/kernel_tests/data_structures:fifo_queue_test PASSED in 14.1s //tensorflow/python/kernel_tests/data_structures:list_ops_test_cpu PASSED in 22.5s //tensorflow/python/kernel_tests/data_structures:listdiff_op_test PASSED in 9.5s //tensorflow/python/kernel_tests/data_structures:lookup_ops_test PASSED in 27.4s //tensorflow/python/kernel_tests/data_structures:map_ops_test PASSED in 12.4s //tensorflow/python/kernel_tests/data_structures:padding_fifo_queue_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/data_structures:priority_queue_test PASSED in 11.3s //tensorflow/python/kernel_tests/data_structures:stack_ops_test_cpu PASSED in 7.2s //tensorflow/python/kernel_tests/data_structures:stage_op_test_cpu PASSED in 9.5s //tensorflow/python/kernel_tests/distributions:bernoulli_test_cpu PASSED in 14.9s //tensorflow/python/kernel_tests/distributions:bijector_test_cpu PASSED in 19.1s //tensorflow/python/kernel_tests/distributions:categorical_test_cpu PASSED in 9.3s //tensorflow/python/kernel_tests/distributions:dirichlet_multinomial_test_cpu PASSED in 22.2s //tensorflow/python/kernel_tests/distributions:dirichlet_test_cpu PASSED in 14.1s //tensorflow/python/kernel_tests/distributions:exponential_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/distributions:gamma_test_cpu PASSED in 44.1s //tensorflow/python/kernel_tests/distributions:identity_bijector_test_cpu PASSED in 14.3s //tensorflow/python/kernel_tests/distributions:kullback_leibler_test_cpu PASSED in 17.1s //tensorflow/python/kernel_tests/distributions:laplace_test_cpu PASSED in 30.4s //tensorflow/python/kernel_tests/distributions:multinomial_test_cpu PASSED in 8.5s //tensorflow/python/kernel_tests/distributions:normal_test_cpu PASSED in 17.8s //tensorflow/python/kernel_tests/distributions:special_math_test_cpu PASSED in 20.8s //tensorflow/python/kernel_tests/distributions:uniform_test_cpu PASSED in 12.7s //tensorflow/python/kernel_tests/image_ops:attention_ops_test PASSED in 20.3s //tensorflow/python/kernel_tests/image_ops:decode_bmp_op_test PASSED in 10.6s //tensorflow/python/kernel_tests/image_ops:decode_compressed_op_test PASSED in 8.2s //tensorflow/python/kernel_tests/image_ops:decode_image_op_test PASSED in 9.3s //tensorflow/python/kernel_tests/image_ops:decode_jpeg_op_test PASSED in 9.8s //tensorflow/python/kernel_tests/image_ops:decode_png_op_test PASSED in 8.4s //tensorflow/python/kernel_tests/image_ops:decode_raw_op_test PASSED in 10.4s //tensorflow/python/kernel_tests/image_ops:draw_bounding_box_op_test_cpu PASSED in 9.5s //tensorflow/python/kernel_tests/image_ops:extract_image_patches_op_test_cpu PASSED in 8.2s //tensorflow/python/kernel_tests/image_ops:extract_volume_patches_op_test_cpu PASSED in 9.1s //tensorflow/python/kernel_tests/io_ops:checkpoint_ops_test PASSED in 10.8s //tensorflow/python/kernel_tests/io_ops:decode_csv_op_test PASSED in 6.3s //tensorflow/python/kernel_tests/io_ops:io_ops_test PASSED in 7.5s //tensorflow/python/kernel_tests/io_ops:parse_single_example_op_test PASSED in 8.2s //tensorflow/python/kernel_tests/io_ops:parsing_ops_test PASSED in 32.7s //tensorflow/python/kernel_tests/io_ops:reader_ops_test PASSED in 12.8s //tensorflow/python/kernel_tests/io_ops:record_input_test PASSED in 32.3s //tensorflow/python/kernel_tests/io_ops:save_restore_ops_test PASSED in 40.1s //tensorflow/python/kernel_tests/linalg:determinant_op_test_cpu PASSED in 8.1s //tensorflow/python/kernel_tests/linalg:linear_operator_addition_test_cpu PASSED in 16.1s //tensorflow/python/kernel_tests/linalg:linear_operator_algebra_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/linalg:linear_operator_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests/linalg:lu_op_test_cpu PASSED in 9.1s //tensorflow/python/kernel_tests/linalg:matrix_inverse_op_test_cpu PASSED in 10.9s //tensorflow/python/kernel_tests/linalg:matrix_logarithm_op_test PASSED in 56.1s //tensorflow/python/kernel_tests/linalg:matrix_solve_ls_op_test_cpu PASSED in 24.2s //tensorflow/python/kernel_tests/linalg:matrix_solve_op_test_cpu PASSED in 31.3s //tensorflow/python/kernel_tests/linalg:matrix_square_root_op_test_cpu PASSED in 9.0s //tensorflow/python/kernel_tests/linalg:slicing_test_cpu PASSED in 12.9s //tensorflow/python/kernel_tests/linalg/sparse:conjugate_gradient_test_cpu PASSED in 12.5s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_test_cpu PASSED in 7.1s //tensorflow/python/kernel_tests/math_ops:aggregate_ops_test_cpu PASSED in 14.1s //tensorflow/python/kernel_tests/math_ops:argmax_op_test_cpu PASSED in 8.4s //tensorflow/python/kernel_tests/math_ops:banded_triangular_solve_op_test_cpu PASSED in 9.9s //tensorflow/python/kernel_tests/math_ops:basic_gpu_test_cpu PASSED in 9.2s //tensorflow/python/kernel_tests/math_ops:bincount_op_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests/math_ops:bucketize_op_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/math_ops:clip_ops_test PASSED in 8.3s //tensorflow/python/kernel_tests/math_ops:confusion_matrix_test PASSED in 12.0s //tensorflow/python/kernel_tests/math_ops:cross_grad_test_cpu PASSED in 10.5s //tensorflow/python/kernel_tests/math_ops:cumulative_logsumexp_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/math_ops:in_topk_op_test_cpu PASSED in 10.5s //tensorflow/python/kernel_tests/math_ops:reduce_benchmark_test_cpu PASSED in 7.7s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_d9m_test_cpu PASSED in 7.4s //tensorflow/python/kernel_tests/math_ops:sets_test PASSED in 24.4s //tensorflow/python/kernel_tests/math_ops:topk_op_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests/math_ops:zero_division_test_cpu PASSED in 9.1s //tensorflow/python/kernel_tests/nn_ops:betainc_op_test_cpu PASSED in 11.5s //tensorflow/python/kernel_tests/nn_ops:bias_op_test_cpu PASSED in 132.0s //tensorflow/python/kernel_tests/nn_ops:conv1d_test_cpu PASSED in 8.7s //tensorflow/python/kernel_tests/nn_ops:conv1d_transpose_test_cpu PASSED in 8.1s //tensorflow/python/kernel_tests/nn_ops:conv2d_transpose_test_cpu PASSED in 9.7s //tensorflow/python/kernel_tests/nn_ops:conv3d_backprop_filter_v2_grad_test_cpu PASSED in 14.0s //tensorflow/python/kernel_tests/nn_ops:conv3d_transpose_test_cpu PASSED in 16.9s //tensorflow/python/kernel_tests/nn_ops:ctc_decoder_ops_test PASSED in 8.4s //tensorflow/python/kernel_tests/nn_ops:ctc_loss_op_test_cpu PASSED in 65.0s //tensorflow/python/kernel_tests/nn_ops:cudnn_d9m_test_cpu PASSED in 8.3s //tensorflow/python/kernel_tests/nn_ops:cudnn_deterministic_ops_test_cpu PASSED in 7.6s //tensorflow/python/kernel_tests/nn_ops:losses_test PASSED in 44.1s //tensorflow/python/kernel_tests/nn_ops:lrn_op_test_cpu PASSED in 9.1s //tensorflow/python/kernel_tests/nn_ops:morphological_ops_test_cpu PASSED in 12.1s //tensorflow/python/kernel_tests/nn_ops:nth_element_op_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/nn_ops:pool_test_cpu PASSED in 38.1s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_3d_test_cpu PASSED in 21.1s //tensorflow/python/kernel_tests/nn_ops:relu_op_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/nn_ops:softmax_op_test_cpu PASSED in 8.4s //tensorflow/python/kernel_tests/nn_ops:softplus_op_test_cpu PASSED in 9.0s //tensorflow/python/kernel_tests/nn_ops:softsign_op_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/nn_ops:xent_op_d9m_test_cpu PASSED in 182.6s //tensorflow/python/kernel_tests/nn_ops:xent_op_test_cpu PASSED in 9.9s //tensorflow/python/kernel_tests/proto:descriptor_source_test PASSED in 10.2s //tensorflow/python/kernel_tests/proto:encode_proto_op_test PASSED in 9.3s //tensorflow/python/kernel_tests/quantization_ops:quantization_ops_test PASSED in 8.3s //tensorflow/python/kernel_tests/random:candidate_sampler_ops_test PASSED in 18.5s //tensorflow/python/kernel_tests/random:multinomial_op_test_cpu PASSED in 7.9s //tensorflow/python/kernel_tests/random:parameterized_truncated_normal_op_test_cpu PASSED in 14.8s //tensorflow/python/kernel_tests/random:random_crop_test_cpu PASSED in 9.2s //tensorflow/python/kernel_tests/random:random_grad_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests/random:random_ops_test_cpu PASSED in 14.4s //tensorflow/python/kernel_tests/random:random_poisson_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests/random:random_shuffle_queue_test PASSED in 7.5s //tensorflow/python/kernel_tests/random:stateful_random_ops_test_cpu PASSED in 18.2s //tensorflow/python/kernel_tests/signal:mel_ops_test_cpu PASSED in 14.9s //tensorflow/python/kernel_tests/signal:mfcc_ops_test_cpu PASSED in 9.1s //tensorflow/python/kernel_tests/signal:reconstruction_ops_test_cpu PASSED in 16.6s //tensorflow/python/kernel_tests/signal:shape_ops_test_cpu PASSED in 17.9s //tensorflow/python/kernel_tests/sparse_ops:sparse_add_op_test PASSED in 11.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_concat_op_test PASSED in 9.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_conditional_accumulator_test PASSED in 8.9s //tensorflow/python/kernel_tests/sparse_ops:sparse_cross_op_test PASSED in 13.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_matmul_op_test_cpu PASSED in 39.8s //tensorflow/python/kernel_tests/sparse_ops:sparse_reorder_op_test PASSED in 9.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_reshape_op_test PASSED in 10.7s //tensorflow/python/kernel_tests/sparse_ops:sparse_serialization_ops_test PASSED in 9.7s //tensorflow/python/kernel_tests/sparse_ops:sparse_slice_op_test PASSED in 9.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_split_op_test_cpu PASSED in 10.9s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_grad_test_cpu PASSED in 18.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_d9m_test_cpu PASSED in 39.7s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_test_cpu PASSED in 27.1s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensors_map_ops_test PASSED in 9.9s //tensorflow/python/kernel_tests/sparse_ops:sparse_to_dense_op_py_test_cpu PASSED in 9.1s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_d9m_test_cpu PASSED in 100.1s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_test_cpu PASSED in 9.2s //tensorflow/python/kernel_tests/sparse_ops:sparsemask_op_test PASSED in 8.1s //tensorflow/python/kernel_tests/strings_ops:as_string_op_test PASSED in 9.8s //tensorflow/python/kernel_tests/strings_ops:base64_ops_test PASSED in 14.2s //tensorflow/python/kernel_tests/strings_ops:reduce_join_op_test_cpu PASSED in 9.5s //tensorflow/python/kernel_tests/strings_ops:regex_full_match_op_test PASSED in 9.9s //tensorflow/python/kernel_tests/strings_ops:regex_replace_op_test PASSED in 8.8s //tensorflow/python/kernel_tests/strings_ops:string_bytes_split_op_test PASSED in 8.8s //tensorflow/python/kernel_tests/strings_ops:string_format_op_test PASSED in 20.6s //tensorflow/python/kernel_tests/strings_ops:string_join_op_test PASSED in 11.3s //tensorflow/python/kernel_tests/strings_ops:string_length_op_test PASSED in 7.7s //tensorflow/python/kernel_tests/strings_ops:string_lower_op_test PASSED in 20.6s //tensorflow/python/kernel_tests/strings_ops:string_split_op_test PASSED in 11.1s //tensorflow/python/kernel_tests/strings_ops:string_strip_op_test PASSED in 11.5s //tensorflow/python/kernel_tests/strings_ops:string_to_hash_bucket_op_test_cpu PASSED in 7.7s //tensorflow/python/kernel_tests/strings_ops:string_to_number_op_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/strings_ops:string_upper_op_test PASSED in 9.2s //tensorflow/python/kernel_tests/strings_ops:substr_op_test PASSED in 12.1s //tensorflow/python/kernel_tests/strings_ops:unicode_decode_op_test PASSED in 18.1s //tensorflow/python/kernel_tests/strings_ops:unicode_encode_op_test PASSED in 9.2s //tensorflow/python/kernel_tests/strings_ops:unicode_script_op_test PASSED in 9.6s //tensorflow/python/kernel_tests/strings_ops:unicode_transcode_op_test PASSED in 9.4s //tensorflow/python/kernel_tests/strings_ops:unsorted_segment_join_op_test_cpu PASSED in 10.7s //tensorflow/python/kernel_tests/summary_ops:summary_ops_test_cpu PASSED in 18.7s //tensorflow/python/kernel_tests/summary_ops:summary_v1_audio_op_test_cpu PASSED in 7.9s //tensorflow/python/kernel_tests/summary_ops:summary_v1_image_op_test_cpu PASSED in 10.6s //tensorflow/python/kernel_tests/summary_ops:summary_v1_ops_test PASSED in 9.2s //tensorflow/python/kernel_tests/summary_ops:summary_v1_tensor_op_test PASSED in 8.4s //tensorflow/python/kernel_tests/v1_compat_tests:array_ops_test_cpu PASSED in 7.7s //tensorflow/python/kernel_tests/v1_compat_tests:dense_update_ops_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/v1_compat_tests:identity_op_py_test PASSED in 8.8s //tensorflow/python/kernel_tests/v1_compat_tests:scatter_nd_ops_test_cpu PASSED in 7.9s //tensorflow/python/kernel_tests/v1_compat_tests:session_ops_test_cpu PASSED in 7.9s //tensorflow/python/kernel_tests/v1_compat_tests:stack_op_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/variables:dense_update_ops_no_tsan_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests/variables:dense_update_ops_test_cpu PASSED in 9.5s //tensorflow/python/kernel_tests/variables:partitioned_variables_test PASSED in 9.7s //tensorflow/python/kernel_tests/variables:resource_variable_ops_test_cpu PASSED in 59.7s //tensorflow/python/kernel_tests/variables:variable_ops_test_cpu PASSED in 9.3s //tensorflow/python/kernel_tests/variables:variable_scope_test PASSED in 34.6s //tensorflow/python/kernel_tests/variables:variables_test PASSED in 12.4s //tensorflow/python/lib/core:custom_float_test PASSED in 8.5s //tensorflow/python/lib/io:file_io_test PASSED in 14.5s //tensorflow/python/lib/io:tf_record_test PASSED in 12.8s //tensorflow/python/module:module_test PASSED in 9.6s //tensorflow/python/ops/losses:util_test PASSED in 7.4s //tensorflow/python/ops/memory_tests:custom_gradient_memory_test_cpu PASSED in 14.0s //tensorflow/python/ops/numpy_ops:np_array_ops_test_cpu PASSED in 65.1s //tensorflow/python/ops/numpy_ops:np_arrays_test_cpu PASSED in 10.3s //tensorflow/python/ops/numpy_ops:np_dtypes_test_cpu PASSED in 8.9s //tensorflow/python/ops/numpy_ops:np_interop_test_cpu PASSED in 40.6s //tensorflow/python/ops/numpy_ops:np_logic_test_cpu PASSED in 12.3s //tensorflow/python/ops/numpy_ops:np_math_ops_test_cpu PASSED in 22.1s //tensorflow/python/ops/numpy_ops:np_random_test_cpu PASSED in 62.3s //tensorflow/python/ops/numpy_ops:np_utils_test_cpu PASSED in 7.8s //tensorflow/python/ops/numpy_ops/integration_test:np_config_test_cpu PASSED in 19.8s //tensorflow/python/ops/numpy_ops/integration_test:public_symbol_test PASSED in 16.5s //tensorflow/python/ops/parallel_for:array_test_cpu PASSED in 40.3s //tensorflow/python/ops/parallel_for:gradients_test_cpu PASSED in 43.9s //tensorflow/python/ops/parallel_for:xla_control_flow_ops_test_cpu PASSED in 59.5s //tensorflow/python/ops/ragged:convert_to_tensor_or_ragged_tensor_op_test PASSED in 8.3s //tensorflow/python/ops/ragged:ragged_batch_gather_op_test PASSED in 43.5s //tensorflow/python/ops/ragged:ragged_bitcast_op_test PASSED in 8.4s //tensorflow/python/ops/ragged:ragged_boolean_mask_op_test PASSED in 14.9s //tensorflow/python/ops/ragged:ragged_concat_op_test PASSED in 10.5s //tensorflow/python/ops/ragged:ragged_const_op_test PASSED in 8.1s //tensorflow/python/ops/ragged:ragged_constant_value_op_test PASSED in 6.3s //tensorflow/python/ops/ragged:ragged_cross_op_test PASSED in 24.4s //tensorflow/python/ops/ragged:ragged_dispatch_test PASSED in 127.4s //tensorflow/python/ops/ragged:ragged_dynamic_partition_op_test_cpu PASSED in 18.4s //tensorflow/python/ops/ragged:ragged_eager_test PASSED in 7.8s //tensorflow/python/ops/ragged:ragged_expand_dims_op_test PASSED in 7.5s //tensorflow/python/ops/ragged:ragged_factory_ops_test_cpu PASSED in 20.2s //tensorflow/python/ops/ragged:ragged_from_sparse_op_test PASSED in 9.8s //tensorflow/python/ops/ragged:ragged_from_tensor_op_test PASSED in 26.7s //tensorflow/python/ops/ragged:ragged_gather_nd_op_test PASSED in 12.3s //tensorflow/python/ops/ragged:ragged_map_flat_values_op_test PASSED in 9.1s //tensorflow/python/ops/ragged:ragged_map_fn_op_test PASSED in 16.6s //tensorflow/python/ops/ragged:ragged_math_ops_test PASSED in 14.2s //tensorflow/python/ops/ragged:ragged_matmul_op_test PASSED in 34.8s //tensorflow/python/ops/ragged:ragged_merge_dims_op_test PASSED in 24.5s //tensorflow/python/ops/ragged:ragged_one_hot_op_test PASSED in 15.5s //tensorflow/python/ops/ragged:ragged_operators_test PASSED in 27.5s //tensorflow/python/ops/ragged:ragged_placeholder_op_test PASSED in 8.4s //tensorflow/python/ops/ragged:ragged_print_op_test PASSED in 15.7s //tensorflow/python/ops/ragged:ragged_range_op_test PASSED in 8.7s //tensorflow/python/ops/ragged:ragged_rank_op_test PASSED in 8.0s //tensorflow/python/ops/ragged:ragged_reduce_op_test PASSED in 32.4s //tensorflow/python/ops/ragged:ragged_resize_image_op_test PASSED in 19.3s //tensorflow/python/ops/ragged:ragged_reverse_op_test PASSED in 8.0s //tensorflow/python/ops/ragged:ragged_row_lengths_op_test PASSED in 8.0s //tensorflow/python/ops/ragged:ragged_row_splits_to_segment_ids_op_test PASSED in 8.0s //tensorflow/python/ops/ragged:ragged_segment_ids_to_row_splits_op_test PASSED in 8.5s //tensorflow/python/ops/ragged:ragged_segment_op_test PASSED in 13.5s //tensorflow/python/ops/ragged:ragged_size_op_test PASSED in 8.2s //tensorflow/python/ops/ragged:ragged_split_op_test PASSED in 42.0s //tensorflow/python/ops/ragged:ragged_squeeze_op_test PASSED in 17.0s //tensorflow/python/ops/ragged:ragged_stack_op_test PASSED in 11.9s //tensorflow/python/ops/ragged:ragged_tensor_bounding_shape_op_test PASSED in 10.8s //tensorflow/python/ops/ragged:ragged_tensor_shape_test PASSED in 67.0s //tensorflow/python/ops/ragged:ragged_tile_op_test PASSED in 42.3s //tensorflow/python/ops/ragged:ragged_to_sparse_op_test PASSED in 10.0s //tensorflow/python/ops/ragged:ragged_to_tensor_op_test PASSED in 63.1s //tensorflow/python/ops/ragged:ragged_util_test PASSED in 23.0s //tensorflow/python/ops/ragged:ragged_where_op_test PASSED in 22.8s //tensorflow/python/ops/ragged:row_partition_test PASSED in 25.3s //tensorflow/python/ops/ragged:string_ngrams_op_test PASSED in 7.9s //tensorflow/python/ops/ragged:strings_reduce_join_op_test PASSED in 10.0s //tensorflow/python/ops/structured:structured_array_ops_test PASSED in 44.1s //tensorflow/python/ops/structured:structured_tensor_slice_test PASSED in 58.0s //tensorflow/python/ops/structured:structured_tensor_spec_test PASSED in 9.2s //tensorflow/python/ops/structured:structured_tensor_test PASSED in 43.4s //tensorflow/python/ops/v1_compat_tests:gradient_checker_test_cpu PASSED in 9.6s //tensorflow/python/platform:benchmark_test PASSED in 9.4s //tensorflow/python/platform:build_info_test PASSED in 17.7s //tensorflow/python/platform:resource_loader_test PASSED in 2.2s //tensorflow/python/profiler:pprof_profiler_test PASSED in 7.5s //tensorflow/python/profiler:profile_context_test_cpu PASSED in 29.0s //tensorflow/python/profiler:profiler_client_test_cpu PASSED in 8.7s //tensorflow/python/profiler:profiler_test_cpu PASSED in 18.9s //tensorflow/python/profiler:profiler_v2_test_cpu PASSED in 7.8s //tensorflow/python/profiler:profiler_wrapper_test PASSED in 7.3s //tensorflow/python/profiler:tfprof_logger_test PASSED in 7.8s //tensorflow/python/profiler/integration_test:profiler_api_test_cpu PASSED in 26.0s //tensorflow/python/profiler/internal:flops_registry_test PASSED in 7.5s //tensorflow/python/profiler/internal:print_model_analysis_test PASSED in 8.7s //tensorflow/python/profiler/internal:run_metadata_test_cpu PASSED in 14.7s //tensorflow/python/saved_model:fingerprinting_test PASSED in 10.1s //tensorflow/python/saved_model:keras_injection_test PASSED in 20.7s //tensorflow/python/saved_model:load_v1_in_v2_test PASSED in 18.4s //tensorflow/python/saved_model:loader_test PASSED in 13.7s //tensorflow/python/saved_model:method_name_updater_test PASSED in 7.6s //tensorflow/python/saved_model:metrics_test PASSED in 10.5s //tensorflow/python/saved_model:nested_structure_coder_test PASSED in 8.8s //tensorflow/python/saved_model:pywrap_saved_model_fingerprinting_test PASSED in 11.6s //tensorflow/python/saved_model:pywrap_saved_model_metrics_test PASSED in 7.0s //tensorflow/python/saved_model:revived_types_test PASSED in 9.4s //tensorflow/python/saved_model:save_context_test PASSED in 10.7s //tensorflow/python/saved_model:save_test PASSED in 24.0s //tensorflow/python/saved_model:saved_model_test PASSED in 21.2s //tensorflow/python/saved_model:signature_def_utils_test PASSED in 12.9s //tensorflow/python/saved_model:simple_save_test PASSED in 7.4s //tensorflow/python/saved_model:tracing_utils_test PASSED in 9.1s //tensorflow/python/saved_model:utils_test PASSED in 9.3s //tensorflow/python/saved_model/model_utils:export_output_test PASSED in 13.4s //tensorflow/python/saved_model/model_utils:export_test PASSED in 14.4s //tensorflow/python/saved_model/model_utils:mode_keys_test PASSED in 7.9s //tensorflow/python/saved_model/registration:registration_saving_test PASSED in 19.1s //tensorflow/python/saved_model/registration:registration_test PASSED in 8.2s //tensorflow/python/saved_model/registration:tf_registration_test PASSED in 21.9s //tensorflow/python/summary:plugin_asset_test PASSED in 13.3s //tensorflow/python/summary:summary_iterator_test PASSED in 10.4s //tensorflow/python/summary:summary_test PASSED in 8.7s //tensorflow/python/summary:summary_v2_test PASSED in 10.6s //tensorflow/python/summary/writer:writer_test PASSED in 21.3s //tensorflow/python/tools:aot_compiled_test PASSED in 19.1s //tensorflow/python/tools:freeze_graph_test PASSED in 17.6s //tensorflow/python/tools:optimize_for_inference_test PASSED in 7.5s //tensorflow/python/tools:print_selective_registration_header_test PASSED in 17.9s //tensorflow/python/tools:saved_model_cli_test PASSED in 23.8s //tensorflow/python/tools:saved_model_utils_test PASSED in 9.7s //tensorflow/python/tools:strip_unused_test PASSED in 7.9s //tensorflow/python/tools/api/generator:create_python_api_test PASSED in 14.8s //tensorflow/python/tools/api/generator:output_init_files_test PASSED in 14.1s //tensorflow/python/tools/api/generator:tensorflow_doc_srcs_test PASSED in 14.7s //tensorflow/python/tpu:bfloat16_test PASSED in 17.2s //tensorflow/python/tpu:feature_column_test PASSED in 12.0s //tensorflow/python/tpu:topology_test PASSED in 9.2s //tensorflow/python/tpu:tpu_embedding_for_serving_test PASSED in 11.4s //tensorflow/python/tpu:tpu_embedding_v2_utils_test PASSED in 9.5s //tensorflow/python/tpu:tpu_infeed_test PASSED in 18.1s //tensorflow/python/tpu:tpu_sharding_test PASSED in 9.6s //tensorflow/python/tpu:tpu_test_wrapper_test PASSED in 7.5s //tensorflow/python/tpu/client:client_py_test PASSED in 8.6s //tensorflow/python/trackable:autotrackable_test PASSED in 21.1s //tensorflow/python/trackable:base_delegate_test PASSED in 11.0s //tensorflow/python/trackable:base_test PASSED in 10.0s //tensorflow/python/trackable:data_structures_test PASSED in 15.0s //tensorflow/python/trackable:python_state_test PASSED in 11.2s //tensorflow/python/trackable:resource_test PASSED in 10.3s //tensorflow/python/trackable:trackable_utils_test PASSED in 8.6s //tensorflow/python/training:adadelta_test_cpu PASSED in 16.3s //tensorflow/python/training:adagrad_da_test_cpu PASSED in 9.0s //tensorflow/python/training:adagrad_test_cpu PASSED in 12.5s //tensorflow/python/training:adam_test_cpu PASSED in 28.0s //tensorflow/python/training:basic_loops_test_cpu PASSED in 8.3s //tensorflow/python/training:basic_session_run_hooks_test PASSED in 21.4s //tensorflow/python/training:checkpoint_ops_test PASSED in 9.9s //tensorflow/python/training:coordinator_test_cpu PASSED in 18.6s //tensorflow/python/training:device_setter_test_cpu PASSED in 11.4s //tensorflow/python/training:ftrl_test_cpu PASSED in 26.1s //tensorflow/python/training:gradient_descent_test_cpu PASSED in 14.6s //tensorflow/python/training:input_test PASSED in 26.6s //tensorflow/python/training:momentum_test_cpu PASSED in 14.8s //tensorflow/python/training:monitored_session_test PASSED in 27.4s //tensorflow/python/training:moving_averages_test_cpu PASSED in 16.2s //tensorflow/python/training:optimizer_test_cpu PASSED in 13.1s //tensorflow/python/training:proximal_adagrad_test_cpu PASSED in 9.6s //tensorflow/python/training:proximal_gradient_descent_test_cpu PASSED in 10.2s //tensorflow/python/training:quantize_training_test_cpu PASSED in 8.4s //tensorflow/python/training:queue_runner_test_cpu PASSED in 8.5s //tensorflow/python/training:rmsprop_test_cpu PASSED in 32.2s //tensorflow/python/training:saver_large_partitioned_variable_test PASSED in 16.6s //tensorflow/python/training:saver_test_2gpu PASSED in 34.3s //tensorflow/python/training:saver_test_cpu PASSED in 35.3s //tensorflow/python/training:server_lib_multiple_containers_test PASSED in 8.5s //tensorflow/python/training:server_lib_same_variables_clear_container_test PASSED in 10.1s //tensorflow/python/training:server_lib_same_variables_clear_test PASSED in 14.3s //tensorflow/python/training:server_lib_same_variables_no_clear_test PASSED in 11.9s //tensorflow/python/training:server_lib_sparse_job_test PASSED in 8.8s //tensorflow/python/training:server_lib_test PASSED in 16.7s //tensorflow/python/training:session_manager_test_cpu PASSED in 77.4s //tensorflow/python/training:slot_creator_test_cpu PASSED in 9.0s //tensorflow/python/training:supervisor_test PASSED in 14.9s //tensorflow/python/training:training_ops_mlir_test_cpu PASSED in 8.8s //tensorflow/python/training:training_ops_test_cpu PASSED in 17.6s //tensorflow/python/training:training_util_test PASSED in 10.1s //tensorflow/python/training:warm_starting_util_test PASSED in 23.4s //tensorflow/python/training/experimental:loss_scale_optimizer_test PASSED in 16.7s //tensorflow/python/training/experimental:loss_scale_test PASSED in 25.3s //tensorflow/python/training/experimental:mixed_precision_test_cpu PASSED in 10.1s //tensorflow/python/training/saving:saveable_object_util_test PASSED in 8.6s //tensorflow/python/util:compat_test PASSED in 10.9s //tensorflow/python/util:decorator_utils_test PASSED in 8.7s //tensorflow/python/util:deprecation_test PASSED in 9.7s //tensorflow/python/util:dispatch_test PASSED in 13.0s //tensorflow/python/util:example_parser_configuration_test PASSED in 8.3s //tensorflow/python/util:fast_module_type_test PASSED in 8.0s //tensorflow/python/util:function_parameter_canonicalizer_test PASSED in 8.8s //tensorflow/python/util:function_utils_test PASSED in 7.8s //tensorflow/python/util:keyword_args_test PASSED in 9.1s //tensorflow/python/util:lock_util_test PASSED in 10.7s //tensorflow/python/util:module_wrapper_test PASSED in 12.6s //tensorflow/python/util:nest_test PASSED in 13.8s //tensorflow/python/util:object_identity_test PASSED in 8.4s //tensorflow/python/util:serialization_test PASSED in 8.1s //tensorflow/python/util:tf_contextlib_test PASSED in 9.2s //tensorflow/python/util:tf_decorator_test PASSED in 11.1s //tensorflow/python/util:tf_export_test PASSED in 6.5s //tensorflow/python/util:tf_inspect_test PASSED in 7.2s //tensorflow/python/util:tf_should_use_test PASSED in 10.6s //tensorflow/python/util:tf_stack_test PASSED in 8.9s //tensorflow/python/util:traceback_utils_test PASSED in 10.9s //tensorflow/python/util:type_annotations_test PASSED in 8.0s //tensorflow/python/util:variable_utils_test PASSED in 8.2s //tensorflow/tools/api/tests:module_test PASSED in 19.1s //tensorflow/tools/benchmark:benchmark_model_test PASSED in 1.8s //tensorflow/tools/common:public_api_test PASSED in 2.9s //tensorflow/tools/common:traverse_test PASSED in 2.0s //tensorflow/tools/compatibility:all_renames_v2_test PASSED in 7.7s //tensorflow/tools/compatibility:ast_edits_test PASSED in 6.3s //tensorflow/tools/compatibility:test_file_v1_0 PASSED in 28.9s //tensorflow/tools/compatibility:test_file_v2_0 PASSED in 18.3s //tensorflow/tools/compatibility:tf_upgrade_test PASSED in 10.0s //tensorflow/tools/compatibility:tf_upgrade_v2_safety_test PASSED in 10.7s //tensorflow/tools/docs:tf_doctest_test PASSED in 0.8s //tensorflow/tools/graph_transforms:file_utils_test PASSED in 0.5s //tensorflow/tools/graph_transforms:transform_graph_test PASSED in 2.0s //tensorflow/tools/graph_transforms:transform_utils_test PASSED in 2.5s //tensorflow/tools/graph_transforms:transforms_test PASSED in 3.2s //tensorflow/tools/proto_text:gen_proto_text_functions_lib_test PASSED in 0.4s //tensorflow/tools/tensorflow_builder/compat_checker:compat_checker_test PASSED in 0.3s //tensorflow/tsl/c:tsl_status_helper_test PASSED in 0.6s //tensorflow/tsl/c:tsl_status_test PASSED in 0.1s //tensorflow/tsl/concurrency:async_value_ref_test PASSED in 0.1s //tensorflow/tsl/concurrency:async_value_test PASSED in 0.1s //tensorflow/tsl/concurrency:concurrent_vector_test PASSED in 0.1s //tensorflow/tsl/cuda:cudnn_version_test PASSED in 0.1s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_agent_test PASSED in 12.9s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_error_util_test PASSED in 0.3s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_recoverable_job_test PASSED in 0.2s //tensorflow/tsl/distributed_runtime/preemption:preemption_notifier_test PASSED in 7.1s //tensorflow/tsl/distributed_runtime/preemption:preemption_sync_manager_test PASSED in 5.8s //tensorflow/tsl/distributed_runtime/rpc:grpc_channel_test PASSED in 0.6s //tensorflow/tsl/distributed_runtime/rpc:grpc_util_test PASSED in 0.2s //tensorflow/tsl/framework:cancellation_test PASSED in 1.2s //tensorflow/tsl/framework/convolution:spatial_convolutions_test PASSED in 0.1s //tensorflow/tsl/lib/gtl:tsl_lib_gtl_tests PASSED in 0.1s //tensorflow/tsl/lib/hash:crc32c_test PASSED in 0.1s //tensorflow/tsl/lib/histogram:histogram_test PASSED in 0.7s //tensorflow/tsl/lib/io:buffered_inputstream_test PASSED in 0.6s //tensorflow/tsl/lib/io:cache_test PASSED in 0.2s //tensorflow/tsl/lib/io:inputbuffer_test PASSED in 1.3s //tensorflow/tsl/lib/io:inputstream_interface_test PASSED in 0.3s //tensorflow/tsl/lib/io:random_inputstream_test PASSED in 0.3s //tensorflow/tsl/lib/io:record_reader_writer_test PASSED in 1.9s //tensorflow/tsl/lib/io:recordio_test PASSED in 0.4s //tensorflow/tsl/lib/io:table_test PASSED in 4.5s //tensorflow/tsl/lib/io:zlib_buffers_test PASSED in 20.8s //tensorflow/tsl/lib/io/snappy:snappy_test PASSED in 0.3s //tensorflow/tsl/lib/math:math_util_test PASSED in 0.1s //tensorflow/tsl/lib/random:distribution_sampler_test PASSED in 0.1s //tensorflow/tsl/lib/random:philox_random_test PASSED in 0.2s //tensorflow/tsl/lib/random:random_distributions_test PASSED in 20.0s //tensorflow/tsl/lib/random:simple_philox_test PASSED in 0.4s //tensorflow/tsl/lib/random:weighted_picker_test PASSED in 11.5s //tensorflow/tsl/platform:ctstring_test PASSED in 0.1s //tensorflow/tsl/platform:denormal_test PASSED in 0.1s //tensorflow/tsl/platform:errors_test PASSED in 0.1s //tensorflow/tsl/platform:fingerprint_test PASSED in 0.1s //tensorflow/tsl/platform:float8_test PASSED in 0.7s //tensorflow/tsl/platform:hash_test PASSED in 0.4s //tensorflow/tsl/platform:integral_types_test PASSED in 0.2s //tensorflow/tsl/platform:intrusive_ptr_test PASSED in 0.1s //tensorflow/tsl/platform:logging_test PASSED in 25.2s //tensorflow/tsl/platform:mutex_test PASSED in 0.2s //tensorflow/tsl/platform:net_test PASSED in 0.1s //tensorflow/tsl/platform:numbers_test PASSED in 0.3s //tensorflow/tsl/platform:path_test PASSED in 0.1s //tensorflow/tsl/platform:port_test PASSED in 8.1s //tensorflow/tsl/platform:random_test PASSED in 2.4s //tensorflow/tsl/platform:refcount_test PASSED in 0.4s //tensorflow/tsl/platform:retrying_file_system_test PASSED in 0.1s //tensorflow/tsl/platform:retrying_utils_test PASSED in 0.3s //tensorflow/tsl/platform:scanner_test PASSED in 0.2s //tensorflow/tsl/platform:setround_test PASSED in 0.2s //tensorflow/tsl/platform:stacktrace_handler_test PASSED in 2.0s //tensorflow/tsl/platform:stacktrace_test PASSED in 0.1s //tensorflow/tsl/platform:status_matchers_test PASSED in 0.1s //tensorflow/tsl/platform:status_test PASSED in 0.1s //tensorflow/tsl/platform:statusor_test PASSED in 16.1s //tensorflow/tsl/platform:str_util_test PASSED in 0.4s //tensorflow/tsl/platform:strcat_test PASSED in 0.1s //tensorflow/tsl/platform:stringpiece_test PASSED in 0.1s //tensorflow/tsl/platform:stringprintf_test PASSED in 0.1s //tensorflow/tsl/platform:subprocess_test PASSED in 0.1s //tensorflow/tsl/platform:tstring_test PASSED in 0.1s //tensorflow/tsl/platform:unbounded_work_queue_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:compute_engine_metadata_client_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:compute_engine_zone_provider_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:curl_http_request_test PASSED in 9.5s //tensorflow/tsl/platform/cloud:expiring_lru_cache_test PASSED in 0.2s //tensorflow/tsl/platform/cloud:gcs_dns_cache_test PASSED in 0.3s //tensorflow/tsl/platform/cloud:gcs_file_system_test PASSED in 5.1s //tensorflow/tsl/platform/cloud:gcs_throttle_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:google_auth_provider_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:oauth_client_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:ram_file_block_cache_test PASSED in 2.5s //tensorflow/tsl/platform/cloud:time_util_test PASSED in 0.6s //tensorflow/tsl/profiler/backends/cpu:traceme_recorder_test PASSED in 0.4s //tensorflow/tsl/profiler/convert:trace_events_to_json_test PASSED in 0.1s //tensorflow/tsl/profiler/convert:xla_op_utils_test PASSED in 0.6s //tensorflow/tsl/profiler/convert:xplane_to_trace_events_test PASSED in 0.1s //tensorflow/tsl/profiler/lib:profiler_factory_test PASSED in 0.1s //tensorflow/tsl/profiler/lib:profiler_lock_test PASSED in 0.3s //tensorflow/tsl/profiler/lib:scoped_annotation_test PASSED in 0.3s //tensorflow/tsl/profiler/lib:traceme_encode_test PASSED in 0.1s //tensorflow/tsl/profiler/rpc/client:profiler_client_test PASSED in 3.5s //tensorflow/tsl/profiler/rpc/client:remote_profiler_session_manager_test PASSED in 3.4s //tensorflow/tsl/profiler/utils:buffer_pool_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:group_events_test PASSED in 0.2s //tensorflow/tsl/profiler/utils:parse_annotation_test PASSED in 0.2s //tensorflow/tsl/profiler/utils:tf_op_utils_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:timespan_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:tpu_xplane_utils_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:xplane_builder_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:xplane_utils_test PASSED in 0.1s //tensorflow/tsl/util:device_name_utils_test PASSED in 0.2s //tensorflow/tsl/util:stats_calculator_test PASSED in 0.1s //tensorflow/compiler/tests:complex_div_test_cpu PASSED in 13.4s Stats over 2 runs: max = 13.4s, min = 12.5s, avg = 13.0s, dev = 0.4s //tensorflow/compiler/tests:complex_div_test_cpu_mlir_bridge_test PASSED in 7.2s Stats over 2 runs: max = 7.2s, min = 6.2s, avg = 6.7s, dev = 0.5s //tensorflow/compiler/xla/tests:conditional_test_cpu PASSED in 8.7s Stats over 2 runs: max = 8.7s, min = 8.6s, avg = 8.7s, dev = 0.0s //tensorflow/python:control_flow_ops_test_cpu PASSED in 36.8s Stats over 2 runs: max = 36.8s, min = 26.5s, avg = 31.7s, dev = 5.2s //tensorflow/python/data/experimental/kernel_tests/optimization:optimization_test PASSED in 28.5s Stats over 2 runs: max = 28.5s, min = 21.6s, avg = 25.1s, dev = 3.5s //tensorflow/python/data/experimental/kernel_tests/service:metadata_test PASSED in 17.1s Stats over 2 runs: max = 17.1s, min = 15.0s, avg = 16.0s, dev = 1.0s //tensorflow/python/data/kernel_tests:padded_batch_test PASSED in 26.4s Stats over 2 runs: max = 26.4s, min = 23.5s, avg = 25.0s, dev = 1.4s //tensorflow/python/data/kernel_tests:repeat_test PASSED in 40.9s Stats over 2 runs: max = 40.9s, min = 39.2s, avg = 40.0s, dev = 0.8s //tensorflow/python/data/kernel_tests:window_test PASSED in 53.4s Stats over 2 runs: max = 53.4s, min = 44.0s, avg = 48.7s, dev = 4.7s //tensorflow/python/distribute:strategy_common_test_2gpu PASSED in 30.3s Stats over 2 runs: max = 30.3s, min = 25.7s, avg = 28.0s, dev = 2.3s //tensorflow/python/distribute:strategy_common_test_cpu PASSED in 26.6s Stats over 2 runs: max = 26.6s, min = 20.9s, avg = 23.7s, dev = 2.9s //tensorflow/python/distribute:strategy_common_test_xla_2gpu PASSED in 17.8s Stats over 2 runs: max = 17.8s, min = 15.8s, avg = 16.8s, dev = 1.0s //tensorflow/python/kernel_tests/array_ops:scatter_nd_ops_test_cpu PASSED in 13.8s Stats over 2 runs: max = 13.8s, min = 13.8s, avg = 13.8s, dev = 0.0s //tensorflow/python/kernel_tests/array_ops:scatter_ops_test_cpu PASSED in 55.1s Stats over 2 runs: max = 55.1s, min = 54.7s, avg = 54.9s, dev = 0.2s //tensorflow/python/kernel_tests/control_flow:functional_ops_test_cpu PASSED in 16.6s Stats over 2 runs: max = 16.6s, min = 15.8s, avg = 16.2s, dev = 0.4s //tensorflow/python/kernel_tests/control_flow:map_fn_test_cpu PASSED in 8.6s Stats over 2 runs: max = 8.6s, min = 8.0s, avg = 8.3s, dev = 0.3s //tensorflow/python/kernel_tests/nn_ops:atrous_conv2d_test_cpu PASSED in 30.5s Stats over 2 runs: max = 30.5s, min = 21.0s, avg = 25.7s, dev = 4.8s //tensorflow/python/kernel_tests/nn_ops:bias_op_d9m_test_cpu PASSED in 99.1s Stats over 2 runs: max = 99.1s, min = 34.3s, avg = 66.7s, dev = 32.4s //tensorflow/python/kernel_tests/nn_ops:conv2d_backprop_filter_grad_test_cpu PASSED in 66.8s Stats over 2 runs: max = 66.8s, min = 12.2s, avg = 39.5s, dev = 27.3s //tensorflow/compiler/xla/pjrt/distributed:client_server_test FLAKY, failed in 1 out of 2 in 41.9s Stats over 2 runs: max = 41.9s, min = 13.7s, avg = 27.8s, dev = 14.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/pjrt/distributed/client_server_test/test_attempts/attempt_1.log //tensorflow/compiler/tests:spacetobatch_op_test_cpu PASSED in 9.6s Stats over 3 runs: max = 9.6s, min = 9.0s, avg = 9.3s, dev = 0.3s //tensorflow/compiler/tests:spacetobatch_op_test_cpu_mlir_bridge_test PASSED in 9.1s Stats over 3 runs: max = 9.1s, min = 8.2s, avg = 8.7s, dev = 0.3s //tensorflow/compiler/xla/tests:triangular_solve_test_cpu PASSED in 57.3s Stats over 3 runs: max = 57.3s, min = 54.2s, avg = 56.0s, dev = 1.3s //tensorflow/core/data/service:thread_safe_buffer_test PASSED in 0.3s Stats over 3 runs: max = 0.3s, min = 0.2s, avg = 0.2s, dev = 0.0s //tensorflow/python/data/experimental/kernel_tests/service:multi_process_cluster_test PASSED in 25.6s Stats over 3 runs: max = 25.6s, min = 20.6s, avg = 23.7s, dev = 2.2s //tensorflow/python/data/kernel_tests:unique_test PASSED in 26.1s Stats over 3 runs: max = 26.1s, min = 15.1s, avg = 19.0s, dev = 5.1s //tensorflow/python/kernel_tests/array_ops:gather_op_test_cpu PASSED in 58.7s Stats over 3 runs: max = 58.7s, min = 34.3s, avg = 43.2s, dev = 11.0s //tensorflow/python/kernel_tests/array_ops:weights_broadcast_test PASSED in 12.0s Stats over 3 runs: max = 12.0s, min = 11.7s, avg = 11.8s, dev = 0.1s //tensorflow/python/kernel_tests/distributions:util_test_cpu PASSED in 13.1s Stats over 3 runs: max = 13.1s, min = 11.1s, avg = 12.0s, dev = 0.8s //tensorflow/python/kernel_tests/linalg:matrix_triangular_solve_op_test_cpu PASSED in 119.3s Stats over 3 runs: max = 119.3s, min = 8.8s, avg = 45.9s, dev = 51.9s //tensorflow/python/kernel_tests/random:multinomial_op_big_test_cpu PASSED in 15.4s Stats over 3 runs: max = 15.4s, min = 11.7s, avg = 13.0s, dev = 1.7s //tensorflow/python/kernel_tests/proto:decode_proto_op_test FAILED in 3 out of 3 in 8.5s Stats over 3 runs: max = 8.5s, min = 3.5s, avg = 5.4s, dev = 2.2s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/kernel_tests/proto/decode_proto_op_test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/kernel_tests/proto/decode_proto_op_test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/kernel_tests/proto/decode_proto_op_test/test_attempts/attempt_2.log //tensorflow/python/util/protobuf:protobuf_compare_test FAILED in 3 out of 3 in 3.5s Stats over 3 runs: max = 3.5s, min = 2.2s, avg = 2.6s, dev = 0.6s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/util/protobuf/protobuf_compare_test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/util/protobuf/protobuf_compare_test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/util/protobuf/protobuf_compare_test/test_attempts/attempt_2.log //tensorflow/tools/api/tests:api_compatibility_test FAILED in 3 out of 3 in 26.7s Stats over 3 runs: max = 26.7s, min = 12.4s, avg = 17.3s, dev = 6.6s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/tools/api/tests/api_compatibility_test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/tools/api/tests/api_compatibility_test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/tools/api/tests/api_compatibility_test/test_attempts/attempt_2.log //tensorflow/compiler/tests:ternary_ops_test_cpu PASSED in 15.0s Stats over 4 runs: max = 15.0s, min = 10.8s, avg = 12.5s, dev = 1.7s //tensorflow/compiler/tests:ternary_ops_test_cpu_mlir_bridge_test PASSED in 17.0s Stats over 4 runs: max = 17.0s, min = 13.3s, avg = 14.8s, dev = 1.4s //tensorflow/compiler/tests:unary_ops_test_cpu PASSED in 41.2s Stats over 4 runs: max = 41.2s, min = 13.5s, avg = 29.6s, dev = 11.8s //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test PASSED in 41.3s Stats over 4 runs: max = 41.3s, min = 9.5s, avg = 27.8s, dev = 13.7s //tensorflow/compiler/xla/tests:dynamic_ops_test_cpu PASSED in 8.7s Stats over 4 runs: max = 8.7s, min = 7.9s, avg = 8.3s, dev = 0.3s //tensorflow/core/kernels:example_parsing_ops_test PASSED in 1.3s Stats over 4 runs: max = 1.3s, min = 0.9s, avg = 1.2s, dev = 0.2s //tensorflow/python:nn_batchnorm_test_cpu PASSED in 28.5s Stats over 4 runs: max = 28.5s, min = 9.6s, avg = 22.3s, dev = 7.6s //tensorflow/python:nn_fused_batchnorm_d9m_test_cpu PASSED in 17.3s Stats over 4 runs: max = 17.3s, min = 15.3s, avg = 16.2s, dev = 0.8s //tensorflow/python/data/experimental/kernel_tests:auto_shard_dataset_test PASSED in 29.4s Stats over 4 runs: max = 29.4s, min = 15.2s, avg = 23.2s, dev = 5.3s //tensorflow/python/data/experimental/kernel_tests:map_and_batch_test PASSED in 38.3s Stats over 4 runs: max = 38.3s, min = 26.6s, avg = 30.1s, dev = 4.8s //tensorflow/python/data/experimental/kernel_tests:parse_example_dataset_test PASSED in 24.8s Stats over 4 runs: max = 24.8s, min = 13.2s, avg = 18.9s, dev = 5.6s //tensorflow/python/data/experimental/kernel_tests:rebatch_dataset_test PASSED in 25.4s Stats over 4 runs: max = 25.4s, min = 12.4s, avg = 17.1s, dev = 5.0s //tensorflow/python/data/experimental/kernel_tests:sql_dataset_test PASSED in 64.7s Stats over 4 runs: max = 64.7s, min = 54.9s, avg = 59.6s, dev = 4.4s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_ft_test PASSED in 9.4s Stats over 4 runs: max = 9.4s, min = 8.0s, avg = 8.7s, dev = 0.6s //tensorflow/python/data/kernel_tests:batch_test PASSED in 43.0s Stats over 4 runs: max = 43.0s, min = 31.6s, avg = 37.1s, dev = 4.1s //tensorflow/python/data/kernel_tests:fixed_length_record_dataset_test PASSED in 21.3s Stats over 4 runs: max = 21.3s, min = 12.1s, avg = 16.4s, dev = 3.8s //tensorflow/python/data/kernel_tests:from_generator_test PASSED in 19.5s Stats over 4 runs: max = 19.5s, min = 12.1s, avg = 15.6s, dev = 2.8s //tensorflow/python/data/kernel_tests:group_by_window_test PASSED in 18.6s Stats over 4 runs: max = 18.6s, min = 6.8s, avg = 11.6s, dev = 4.8s //tensorflow/python/data/kernel_tests:ragged_batch_test PASSED in 20.8s Stats over 4 runs: max = 20.8s, min = 18.6s, avg = 19.3s, dev = 0.9s //tensorflow/python/data/kernel_tests:shuffle_test PASSED in 38.3s Stats over 4 runs: max = 38.3s, min = 29.1s, avg = 33.3s, dev = 3.7s //tensorflow/python/data/kernel_tests:skip_test PASSED in 28.4s Stats over 4 runs: max = 28.4s, min = 19.7s, avg = 23.5s, dev = 3.7s //tensorflow/python/data/kernel_tests:take_test PASSED in 23.8s Stats over 4 runs: max = 23.8s, min = 22.3s, avg = 23.0s, dev = 0.5s //tensorflow/python/data/kernel_tests:take_while_test PASSED in 32.8s Stats over 4 runs: max = 32.8s, min = 31.1s, avg = 31.9s, dev = 0.7s //tensorflow/python/data/kernel_tests:text_line_dataset_test PASSED in 19.3s Stats over 4 runs: max = 19.3s, min = 12.8s, avg = 16.0s, dev = 2.7s //tensorflow/python/data/kernel_tests:zip_test PASSED in 13.1s Stats over 4 runs: max = 13.1s, min = 11.0s, avg = 11.9s, dev = 0.9s //tensorflow/python/debug/lib:dumping_callback_test_cpu PASSED in 17.2s Stats over 4 runs: max = 17.2s, min = 16.6s, avg = 16.9s, dev = 0.3s //tensorflow/python/distribute:cross_device_ops_test_2gpu PASSED in 35.7s Stats over 4 runs: max = 35.7s, min = 25.4s, avg = 30.7s, dev = 4.4s //tensorflow/python/distribute:cross_device_ops_test_cpu PASSED in 32.3s Stats over 4 runs: max = 32.3s, min = 23.1s, avg = 26.7s, dev = 3.8s //tensorflow/python/distribute:strategy_gather_test_2gpu PASSED in 25.4s Stats over 4 runs: max = 25.4s, min = 14.0s, avg = 19.5s, dev = 5.4s //tensorflow/python/distribute:strategy_gather_test_cpu PASSED in 24.3s Stats over 4 runs: max = 24.3s, min = 13.4s, avg = 18.6s, dev = 5.1s //tensorflow/python/distribute:strategy_gather_test_xla_2gpu PASSED in 23.1s Stats over 4 runs: max = 23.1s, min = 13.6s, avg = 17.9s, dev = 4.2s //tensorflow/python/framework:convert_to_constants_test PASSED in 26.4s Stats over 4 runs: max = 26.4s, min = 17.1s, avg = 20.6s, dev = 3.5s //tensorflow/python/kernel_tests:collective_ops_test_2gpu PASSED in 34.2s Stats over 4 runs: max = 34.2s, min = 32.9s, avg = 33.3s, dev = 0.5s //tensorflow/python/kernel_tests:collective_ops_test_cpu PASSED in 34.7s Stats over 4 runs: max = 34.7s, min = 33.2s, avg = 33.9s, dev = 0.6s //tensorflow/python/kernel_tests/array_ops:concat_op_test_cpu PASSED in 17.0s Stats over 4 runs: max = 17.0s, min = 15.0s, avg = 16.1s, dev = 0.8s //tensorflow/python/kernel_tests/array_ops:init_ops_test_cpu PASSED in 55.5s Stats over 4 runs: max = 55.5s, min = 23.1s, avg = 38.6s, dev = 13.2s //tensorflow/python/kernel_tests/array_ops:split_op_test_cpu PASSED in 30.8s Stats over 4 runs: max = 30.8s, min = 8.6s, avg = 16.5s, dev = 9.0s //tensorflow/python/kernel_tests/linalg:einsum_op_test_cpu PASSED in 85.1s Stats over 4 runs: max = 85.1s, min = 16.2s, avg = 47.1s, dev = 28.2s //tensorflow/python/kernel_tests/linalg:linear_operator_lower_triangular_test_cpu PASSED in 28.9s Stats over 4 runs: max = 28.9s, min = 25.5s, avg = 26.8s, dev = 1.4s //tensorflow/python/kernel_tests/nn_ops:conv_ops_test_cpu PASSED in 58.2s Stats over 4 runs: max = 58.2s, min = 38.6s, avg = 45.3s, dev = 8.0s //tensorflow/python/kernel_tests/random:random_gamma_test_cpu PASSED in 71.3s Stats over 4 runs: max = 71.3s, min = 10.9s, avg = 36.5s, dev = 26.2s //tensorflow/python/kernel_tests/signal:window_ops_test_cpu PASSED in 21.9s Stats over 4 runs: max = 21.9s, min = 21.1s, avg = 21.4s, dev = 0.3s //tensorflow/python/ops/ragged:ragged_gather_op_test PASSED in 103.4s Stats over 4 runs: max = 103.4s, min = 51.1s, avg = 75.9s, dev = 18.7s //tensorflow/python/ops/ragged:ragged_getitem_test PASSED in 72.7s Stats over 4 runs: max = 72.7s, min = 49.6s, avg = 58.8s, dev = 8.5s //tensorflow/tools/docs:tf_doctest PASSED in 62.4s Stats over 4 runs: max = 62.4s, min = 36.1s, avg = 46.7s, dev = 9.7s //tensorflow/compiler/tests:async_comp_test_cpu PASSED in 7.3s Stats over 5 runs: max = 7.3s, min = 3.5s, avg = 5.7s, dev = 1.4s //tensorflow/compiler/tests:conv3d_test_cpu PASSED in 15.7s Stats over 5 runs: max = 15.7s, min = 6.6s, avg = 10.7s, dev = 3.9s //tensorflow/compiler/tests:conv3d_test_cpu_mlir_bridge_test PASSED in 17.2s Stats over 5 runs: max = 17.2s, min = 9.9s, avg = 12.9s, dev = 3.1s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu PASSED in 12.6s Stats over 5 runs: max = 12.6s, min = 7.2s, avg = 9.6s, dev = 2.3s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu_mlir_bridge_test PASSED in 16.5s Stats over 5 runs: max = 16.5s, min = 12.1s, avg = 14.0s, dev = 1.8s //tensorflow/compiler/tests:fused_batchnorm_test_cpu PASSED in 7.4s Stats over 5 runs: max = 7.4s, min = 6.7s, avg = 7.1s, dev = 0.3s //tensorflow/compiler/tests:fused_batchnorm_test_cpu_mlir_bridge_test PASSED in 10.7s Stats over 5 runs: max = 10.7s, min = 9.2s, avg = 10.2s, dev = 0.6s //tensorflow/compiler/tests:image_ops_jit_compile_test_cpu PASSED in 7.3s Stats over 5 runs: max = 7.3s, min = 5.7s, avg = 6.1s, dev = 0.6s //tensorflow/compiler/tests:reduce_ops_test_cpu PASSED in 10.1s Stats over 5 runs: max = 10.1s, min = 8.9s, avg = 9.6s, dev = 0.4s //tensorflow/compiler/tests:reduce_ops_test_cpu_mlir_bridge_test PASSED in 14.9s Stats over 5 runs: max = 14.9s, min = 12.3s, avg = 14.0s, dev = 0.9s //tensorflow/compiler/tests:repeat_op_test_cpu PASSED in 9.0s Stats over 5 runs: max = 9.0s, min = 7.8s, avg = 8.2s, dev = 0.4s //tensorflow/compiler/tests:repeat_op_test_cpu_mlir_bridge_test PASSED in 10.4s Stats over 5 runs: max = 10.4s, min = 8.8s, avg = 9.3s, dev = 0.6s //tensorflow/compiler/tests:special_math_test_cpu PASSED in 130.1s Stats over 5 runs: max = 130.1s, min = 16.0s, avg = 55.1s, dev = 39.9s //tensorflow/compiler/tests:special_math_test_cpu_mlir_bridge_test PASSED in 132.3s Stats over 5 runs: max = 132.3s, min = 28.1s, avg = 67.7s, dev = 35.4s //tensorflow/compiler/xla/client/lib:self_adjoint_eig_test_cpu PASSED in 23.9s Stats over 5 runs: max = 23.9s, min = 12.1s, avg = 19.2s, dev = 5.2s //tensorflow/core/grappler/optimizers:constant_folding_test PASSED in 2.6s Stats over 5 runs: max = 2.6s, min = 1.6s, avg = 2.2s, dev = 0.4s //tensorflow/python/distribute:mirrored_strategy_test_2gpu PASSED in 12.1s Stats over 5 runs: max = 12.1s, min = 10.8s, avg = 11.5s, dev = 0.5s //tensorflow/python/distribute:mirrored_strategy_test_cpu PASSED in 10.1s Stats over 5 runs: max = 10.1s, min = 8.7s, avg = 9.3s, dev = 0.5s //tensorflow/python/distribute:moving_averages_test_2gpu PASSED in 32.8s Stats over 5 runs: max = 32.8s, min = 25.3s, avg = 29.1s, dev = 2.5s //tensorflow/python/distribute:moving_averages_test_cpu PASSED in 18.5s Stats over 5 runs: max = 18.5s, min = 14.8s, avg = 17.1s, dev = 1.2s //tensorflow/python/distribute:vars_test_2gpu PASSED in 14.4s Stats over 5 runs: max = 14.4s, min = 13.0s, avg = 13.8s, dev = 0.5s //tensorflow/python/distribute:vars_test_cpu PASSED in 17.6s Stats over 5 runs: max = 17.6s, min = 15.4s, avg = 16.6s, dev = 0.8s //tensorflow/python/eager:device_placement_test_cpu PASSED in 12.5s Stats over 5 runs: max = 12.5s, min = 11.7s, avg = 12.1s, dev = 0.3s //tensorflow/python/eager:forwardprop_test_cpu PASSED in 85.4s Stats over 5 runs: max = 85.4s, min = 18.5s, avg = 44.7s, dev = 22.3s //tensorflow/python/eager/polymorphic_function:gradients_test_cpu PASSED in 21.9s Stats over 5 runs: max = 21.9s, min = 13.0s, avg = 16.8s, dev = 3.9s //tensorflow/python/kernel_tests/linalg:cholesky_op_test_cpu PASSED in 50.3s Stats over 5 runs: max = 50.3s, min = 31.7s, avg = 41.4s, dev = 6.5s //tensorflow/python/kernel_tests/linalg:linear_operator_adjoint_test_cpu PASSED in 36.5s Stats over 5 runs: max = 36.5s, min = 27.6s, avg = 32.8s, dev = 3.7s //tensorflow/python/kernel_tests/linalg:linear_operator_composition_test_cpu PASSED in 52.1s Stats over 5 runs: max = 52.1s, min = 40.3s, avg = 46.8s, dev = 4.2s //tensorflow/python/kernel_tests/linalg:linear_operator_diag_test_cpu PASSED in 25.4s Stats over 5 runs: max = 25.4s, min = 24.4s, avg = 25.0s, dev = 0.4s //tensorflow/python/kernel_tests/linalg:linear_operator_full_matrix_test_cpu PASSED in 26.5s Stats over 5 runs: max = 26.5s, min = 25.4s, avg = 26.0s, dev = 0.4s //tensorflow/python/kernel_tests/linalg:linear_operator_householder_test_cpu PASSED in 37.1s Stats over 5 runs: max = 37.1s, min = 34.1s, avg = 35.3s, dev = 1.0s //tensorflow/python/kernel_tests/linalg:linear_operator_identity_test_cpu PASSED in 31.4s Stats over 5 runs: max = 31.4s, min = 29.6s, avg = 30.4s, dev = 0.8s //tensorflow/python/kernel_tests/linalg:linear_operator_inversion_test_cpu PASSED in 30.7s Stats over 5 runs: max = 30.7s, min = 28.4s, avg = 29.6s, dev = 0.7s //tensorflow/python/kernel_tests/linalg:linear_operator_permutation_test_cpu PASSED in 18.3s Stats over 5 runs: max = 18.3s, min = 15.9s, avg = 17.5s, dev = 0.9s //tensorflow/python/kernel_tests/linalg:linear_operator_toeplitz_test_cpu PASSED in 16.0s Stats over 5 runs: max = 16.0s, min = 13.4s, avg = 14.3s, dev = 0.9s //tensorflow/python/kernel_tests/linalg:linear_operator_tridiag_test_cpu PASSED in 74.3s Stats over 5 runs: max = 74.3s, min = 71.9s, avg = 73.2s, dev = 0.8s //tensorflow/python/kernel_tests/linalg:linear_operator_util_test_cpu PASSED in 7.6s Stats over 5 runs: max = 7.6s, min = 7.5s, avg = 7.5s, dev = 0.0s //tensorflow/python/kernel_tests/linalg:linear_operator_zeros_test_cpu PASSED in 14.1s Stats over 5 runs: max = 14.1s, min = 13.0s, avg = 13.6s, dev = 0.4s //tensorflow/python/kernel_tests/nn_ops:fractional_avg_pool_op_test PASSED in 16.1s Stats over 5 runs: max = 16.1s, min = 7.3s, avg = 9.7s, dev = 3.3s //tensorflow/python/kernel_tests/nn_ops:fractional_max_pool_op_test PASSED in 17.9s Stats over 5 runs: max = 17.9s, min = 6.6s, avg = 10.0s, dev = 4.1s //tensorflow/python/kernel_tests/sparse_ops:sparse_ops_test_cpu PASSED in 38.8s Stats over 5 runs: max = 38.8s, min = 10.4s, avg = 17.0s, dev = 10.9s //tensorflow/python/ops/parallel_for:math_test_cpu PASSED in 89.6s Stats over 5 runs: max = 89.6s, min = 25.6s, avg = 50.6s, dev = 22.4s //tensorflow/compiler/tests:scan_ops_test_cpu PASSED in 14.3s Stats over 6 runs: max = 14.3s, min = 10.9s, avg = 12.8s, dev = 1.0s //tensorflow/compiler/tests:scan_ops_test_cpu_mlir_bridge_test PASSED in 19.8s Stats over 6 runs: max = 19.8s, min = 12.7s, avg = 16.7s, dev = 2.1s //tensorflow/compiler/xla/tests:prng_test_cpu PASSED in 8.6s Stats over 6 runs: max = 8.6s, min = 7.4s, avg = 8.1s, dev = 0.5s //tensorflow/python:accumulate_n_benchmark_cpu PASSED in 12.6s Stats over 6 runs: max = 12.6s, min = 12.1s, avg = 12.4s, dev = 0.2s //tensorflow/python/data/experimental/kernel_tests:make_batched_features_dataset_test PASSED in 28.2s Stats over 6 runs: max = 28.2s, min = 10.5s, avg = 18.1s, dev = 7.5s //tensorflow/python/kernel_tests/array_ops:diag_op_test_cpu PASSED in 75.4s Stats over 6 runs: max = 75.4s, min = 10.9s, avg = 24.4s, dev = 22.9s //tensorflow/python/kernel_tests/math_ops:reduction_ops_test_cpu PASSED in 37.7s Stats over 6 runs: max = 37.7s, min = 25.7s, avg = 32.3s, dev = 3.8s //tensorflow/python/distribute/experimental/rpc:rpc_ops_test PASSED in 13.6s Stats over 7 runs: max = 13.6s, min = 6.8s, avg = 9.1s, dev = 2.3s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu PASSED in 55.0s Stats over 8 runs: max = 55.0s, min = 6.7s, avg = 22.2s, dev = 16.2s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu_mlir_bridge_test PASSED in 66.7s Stats over 8 runs: max = 66.7s, min = 3.9s, avg = 26.0s, dev = 19.0s //tensorflow/python/data/experimental/kernel_tests:csv_dataset_test PASSED in 27.7s Stats over 8 runs: max = 27.7s, min = 5.0s, avg = 14.4s, dev = 7.9s //tensorflow/python/data/experimental/kernel_tests:parallel_interleave_test PASSED in 33.3s Stats over 8 runs: max = 33.3s, min = 15.5s, avg = 24.1s, dev = 6.2s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_ft_test PASSED in 48.7s Stats over 8 runs: max = 48.7s, min = 14.8s, avg = 30.8s, dev = 13.5s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_test PASSED in 45.9s Stats over 8 runs: max = 45.9s, min = 16.8s, avg = 26.7s, dev = 11.9s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_test PASSED in 18.0s Stats over 8 runs: max = 18.0s, min = 6.5s, avg = 10.9s, dev = 4.0s //tensorflow/python/data/experimental/kernel_tests/service:fault_tolerance_test PASSED in 26.4s Stats over 8 runs: max = 26.4s, min = 17.6s, avg = 19.3s, dev = 2.7s //tensorflow/python/data/kernel_tests:filter_test PASSED in 16.6s Stats over 8 runs: max = 16.6s, min = 13.7s, avg = 15.3s, dev = 0.9s //tensorflow/python/data/kernel_tests:flat_map_test PASSED in 22.2s Stats over 8 runs: max = 22.2s, min = 13.1s, avg = 17.1s, dev = 3.7s //tensorflow/python/data/kernel_tests:shard_test PASSED in 23.5s Stats over 8 runs: max = 23.5s, min = 17.6s, avg = 20.7s, dev = 2.1s //tensorflow/python/data/kernel_tests:tf_record_dataset_test PASSED in 27.9s Stats over 8 runs: max = 27.9s, min = 15.6s, avg = 21.5s, dev = 3.3s //tensorflow/python/distribute/failure_handling:gce_failure_handler_test PASSED in 92.0s Stats over 8 runs: max = 92.0s, min = 22.4s, avg = 43.0s, dev = 28.5s //tensorflow/python/kernel_tests/linalg:linalg_ops_test_cpu PASSED in 52.8s Stats over 8 runs: max = 52.8s, min = 34.2s, avg = 43.6s, dev = 6.5s //tensorflow/python/kernel_tests/linalg:linear_operator_block_diag_test_cpu PASSED in 79.6s Stats over 8 runs: max = 79.6s, min = 49.0s, avg = 68.2s, dev = 10.9s //tensorflow/python/kernel_tests/linalg:linear_operator_block_lower_triangular_test_cpu PASSED in 67.7s Stats over 8 runs: max = 67.7s, min = 39.8s, avg = 53.2s, dev = 10.5s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_d9m_test_cpu PASSED in 64.3s Stats over 8 runs: max = 64.3s, min = 8.3s, avg = 17.8s, dev = 18.1s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_test_cpu PASSED in 7.9s Stats over 8 runs: max = 7.9s, min = 4.9s, avg = 6.4s, dev = 1.0s //tensorflow/python/kernel_tests/signal:fft_ops_test_cpu PASSED in 22.6s Stats over 8 runs: max = 22.6s, min = 10.2s, avg = 14.9s, dev = 4.8s //tensorflow/python/ops/ragged:dynamic_ragged_shape_test PASSED in 52.7s Stats over 8 runs: max = 52.7s, min = 33.3s, avg = 40.0s, dev = 7.2s //tensorflow/python/ops/ragged:ragged_tensor_test PASSED in 27.3s Stats over 8 runs: max = 27.3s, min = 14.6s, avg = 19.3s, dev = 3.6s //tensorflow/python/distribute/failure_handling:failure_handler_test FLAKY, failed in 1 out of 9 in 64.9s Stats over 9 runs: max = 64.9s, min = 22.5s, avg = 43.9s, dev = 13.2s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/failure_handling/failure_handler_test/shard_1_of_8/test_attempts/attempt_1.log //tensorflow/compiler/tests:bincount_op_test_cpu PASSED in 17.2s Stats over 10 runs: max = 17.2s, min = 14.0s, avg = 15.6s, dev = 0.9s //tensorflow/compiler/tests:conv2d_test_cpu PASSED in 11.0s Stats over 10 runs: max = 11.0s, min = 9.7s, avg = 10.1s, dev = 0.4s //tensorflow/compiler/tests:conv2d_test_cpu_mlir_bridge_test PASSED in 8.8s Stats over 10 runs: max = 8.8s, min = 6.9s, avg = 8.1s, dev = 0.6s //tensorflow/compiler/tests:image_ops_test_cpu PASSED in 17.4s Stats over 10 runs: max = 17.4s, min = 12.4s, avg = 14.7s, dev = 1.5s //tensorflow/compiler/tests:random_ops_test_cpu PASSED in 18.7s Stats over 10 runs: max = 18.7s, min = 13.2s, avg = 16.3s, dev = 1.7s //tensorflow/compiler/tests:random_ops_test_cpu_mlir_bridge_test PASSED in 20.7s Stats over 10 runs: max = 20.7s, min = 13.8s, avg = 16.9s, dev = 1.9s //tensorflow/compiler/tests:stateless_random_ops_test_cpu PASSED in 73.4s Stats over 10 runs: max = 73.4s, min = 39.9s, avg = 56.0s, dev = 12.2s //tensorflow/compiler/tests:stateless_random_ops_test_cpu_mlir_bridge_test PASSED in 71.7s Stats over 10 runs: max = 71.7s, min = 32.8s, avg = 52.9s, dev = 13.0s //tensorflow/compiler/xla/client/lib:svd_test_cpu PASSED in 86.8s Stats over 10 runs: max = 86.8s, min = 8.6s, avg = 28.3s, dev = 29.8s //tensorflow/compiler/xla/client/lib:tridiagonal_test_cpu PASSED in 6.9s Stats over 10 runs: max = 6.9s, min = 6.1s, avg = 6.6s, dev = 0.2s //tensorflow/compiler/xla/service/cpu:cpu_runtime_test PASSED in 3.2s Stats over 10 runs: max = 3.2s, min = 0.7s, avg = 2.3s, dev = 0.8s //tensorflow/python:special_math_ops_test_cpu PASSED in 53.2s Stats over 10 runs: max = 53.2s, min = 11.8s, avg = 18.5s, dev = 11.9s //tensorflow/python/data/kernel_tests:rejection_resample_test PASSED in 15.5s Stats over 10 runs: max = 15.5s, min = 8.5s, avg = 11.6s, dev = 2.1s //tensorflow/python/distribute:input_lib_test_2gpu PASSED in 32.0s Stats over 10 runs: max = 32.0s, min = 22.7s, avg = 26.4s, dev = 2.7s //tensorflow/python/distribute:input_lib_test_cpu PASSED in 59.2s Stats over 10 runs: max = 59.2s, min = 50.1s, avg = 53.5s, dev = 2.7s //tensorflow/python/distribute:input_lib_type_spec_test_2gpu PASSED in 24.6s Stats over 10 runs: max = 24.6s, min = 6.9s, avg = 15.6s, dev = 5.6s //tensorflow/python/distribute:input_lib_type_spec_test_cpu PASSED in 16.9s Stats over 10 runs: max = 16.9s, min = 6.2s, avg = 11.4s, dev = 3.5s //tensorflow/python/framework:config_vgpu_test_2gpu PASSED in 8.1s Stats over 10 runs: max = 8.1s, min = 5.8s, avg = 6.9s, dev = 0.7s //tensorflow/python/framework:config_vgpu_test_cpu PASSED in 8.9s Stats over 10 runs: max = 8.9s, min = 5.5s, avg = 7.0s, dev = 1.1s //tensorflow/python/framework:function_test_cpu PASSED in 61.2s Stats over 10 runs: max = 61.2s, min = 6.6s, avg = 13.7s, dev = 16.1s //tensorflow/python/grappler:cluster_test_cpu PASSED in 8.9s Stats over 10 runs: max = 8.9s, min = 5.8s, avg = 7.4s, dev = 1.3s //tensorflow/python/kernel_tests/array_ops:array_ops_test_cpu PASSED in 12.7s Stats over 10 runs: max = 12.7s, min = 8.0s, avg = 9.5s, dev = 1.4s //tensorflow/python/kernel_tests/array_ops:inplace_ops_test_cpu PASSED in 9.5s Stats over 10 runs: max = 9.5s, min = 8.7s, avg = 9.1s, dev = 0.3s //tensorflow/python/kernel_tests/linalg:linear_operator_kronecker_test_cpu PASSED in 33.4s Stats over 10 runs: max = 33.4s, min = 30.8s, avg = 31.9s, dev = 0.9s //tensorflow/python/kernel_tests/linalg:linear_operator_low_rank_update_test_cpu PASSED in 68.9s Stats over 10 runs: max = 68.9s, min = 61.9s, avg = 65.8s, dev = 2.0s //tensorflow/python/kernel_tests/linalg:tridiagonal_matmul_op_test_cpu PASSED in 126.9s Stats over 10 runs: max = 126.9s, min = 13.6s, avg = 25.8s, dev = 33.7s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_ops_test_cpu PASSED in 50.6s Stats over 10 runs: max = 50.6s, min = 6.8s, avg = 26.1s, dev = 14.1s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_test_cpu PASSED in 26.9s Stats over 10 runs: max = 26.9s, min = 8.5s, avg = 15.4s, dev = 6.8s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_test_cpu PASSED in 27.2s Stats over 10 runs: max = 27.2s, min = 8.5s, avg = 13.0s, dev = 6.7s //tensorflow/python/kernel_tests/nn_ops:rnn_test_cpu PASSED in 12.4s Stats over 10 runs: max = 12.4s, min = 7.9s, avg = 10.1s, dev = 1.7s //tensorflow/python/kernel_tests/random:random_index_shuffle_test PASSED in 8.5s Stats over 10 runs: max = 8.5s, min = 4.4s, avg = 6.2s, dev = 1.3s //tensorflow/python/kernel_tests/random:stateless_random_ops_test_cpu PASSED in 94.7s Stats over 10 runs: max = 94.7s, min = 17.2s, avg = 54.9s, dev = 36.2s //tensorflow/python/ops/ragged:ragged_tensor_supported_values_test PASSED in 30.8s Stats over 10 runs: max = 30.8s, min = 21.5s, avg = 25.0s, dev = 2.5s //tensorflow/python/saved_model:load_test_cpu PASSED in 45.5s Stats over 10 runs: max = 45.5s, min = 27.8s, avg = 32.0s, dev = 4.8s //tensorflow/dtensor/python/tests:input_util_test FLAKY, failed in 2 out of 10 in 19.2s Stats over 10 runs: max = 19.2s, min = 13.1s, avg = 16.7s, dev = 2.1s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/dtensor/python/tests/input_util_test/shard_4_of_8/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/dtensor/python/tests/input_util_test/shard_4_of_8/test_attempts/attempt_2.log //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test FAILED in 3 out of 10 in 45.6s Stats over 10 runs: max = 45.6s, min = 6.6s, avg = 15.9s, dev = 13.4s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test/shard_2_of_8/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test/shard_2_of_8/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test/shard_2_of_8/test_attempts/attempt_2.log //tensorflow/python/kernel_tests/data_structures:tensor_array_ops_test_cpu FLAKY, failed in 1 out of 11 in 11.2s Stats over 11 runs: max = 11.2s, min = 5.1s, avg = 8.8s, dev = 1.5s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/kernel_tests/data_structures/tensor_array_ops_test_cpu/shard_2_of_10/test_attempts/attempt_1.log //tensorflow/compiler/tests:fft_test_cpu PASSED in 22.8s Stats over 12 runs: max = 22.8s, min = 8.1s, avg = 13.9s, dev = 5.3s //tensorflow/compiler/xla/service:triangular_solve_expander_test PASSED in 4.6s Stats over 12 runs: max = 4.6s, min = 2.1s, avg = 2.9s, dev = 0.7s //tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 20.3s Stats over 12 runs: max = 20.3s, min = 6.5s, avg = 11.3s, dev = 4.2s //tensorflow/python/data/kernel_tests:choose_from_datasets_test PASSED in 14.9s Stats over 12 runs: max = 14.9s, min = 8.7s, avg = 11.1s, dev = 1.6s //tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu PASSED in 12.4s Stats over 12 runs: max = 12.4s, min = 8.1s, avg = 10.0s, dev = 1.4s //tensorflow/python/distribute:multi_process_runner_test_2gpu PASSED in 225.6s Stats over 12 runs: max = 225.6s, min = 13.7s, avg = 51.0s, dev = 58.6s //tensorflow/python/distribute:multi_process_runner_test_cpu PASSED in 220.1s Stats over 12 runs: max = 220.1s, min = 9.5s, avg = 47.5s, dev = 58.1s //tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 29.4s Stats over 15 runs: max = 29.4s, min = 25.8s, avg = 27.7s, dev = 1.1s //tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 75.0s Stats over 15 runs: max = 75.0s, min = 65.6s, avg = 69.9s, dev = 2.7s //tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu PASSED in 46.7s Stats over 15 runs: max = 46.7s, min = 8.9s, avg = 15.3s, dev = 9.6s //tensorflow/python:image_ops_test_cpu PASSED in 18.9s Stats over 16 runs: max = 18.9s, min = 9.1s, avg = 14.3s, dev = 3.3s //tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 23.3s Stats over 16 runs: max = 23.3s, min = 12.0s, avg = 17.1s, dev = 3.4s //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test PASSED in 27.9s Stats over 16 runs: max = 27.9s, min = 9.2s, avg = 15.0s, dev = 4.7s //tensorflow/python/data/kernel_tests:snapshot_test PASSED in 26.1s Stats over 16 runs: max = 26.1s, min = 12.9s, avg = 18.0s, dev = 4.0s //tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 28.0s Stats over 16 runs: max = 28.0s, min = 7.5s, avg = 10.8s, dev = 4.6s //tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test PASSED in 10.6s Stats over 16 runs: max = 10.6s, min = 4.7s, avg = 7.3s, dev = 1.5s //tensorflow/python/kernel_tests/signal:dct_ops_test_cpu PASSED in 13.6s Stats over 16 runs: max = 13.6s, min = 8.4s, avg = 11.3s, dev = 1.9s //tensorflow/python/ops/parallel_for:control_flow_ops_test_cpu PASSED in 60.8s Stats over 16 runs: max = 60.8s, min = 12.4s, avg = 21.4s, dev = 10.8s //tensorflow/python/data/kernel_tests:map_test PASSED in 38.3s Stats over 19 runs: max = 38.3s, min = 11.3s, avg = 17.3s, dev = 5.8s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu PASSED in 7.1s Stats over 20 runs: max = 7.1s, min = 3.1s, avg = 5.3s, dev = 1.3s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test PASSED in 6.5s Stats over 20 runs: max = 6.5s, min = 4.5s, avg = 5.7s, dev = 0.7s //tensorflow/compiler/tests:pooling_ops_test_cpu PASSED in 8.3s Stats over 20 runs: max = 8.3s, min = 2.7s, avg = 5.0s, dev = 1.5s //tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test PASSED in 11.1s Stats over 20 runs: max = 11.1s, min = 3.3s, avg = 6.7s, dev = 1.9s //tensorflow/compiler/xla/tests:convolution_dimension_numbers_test_cpu PASSED in 7.4s Stats over 20 runs: max = 7.4s, min = 6.0s, avg = 6.5s, dev = 0.4s //tensorflow/compiler/xla/tests:dot_operation_single_threaded_runtime_test_cpu PASSED in 13.8s Stats over 20 runs: max = 13.8s, min = 9.9s, avg = 12.0s, dev = 1.0s //tensorflow/compiler/xla/tests:dot_operation_test_cpu PASSED in 13.4s Stats over 20 runs: max = 13.4s, min = 10.6s, avg = 12.2s, dev = 0.8s //tensorflow/compiler/xla/tests:reduce_window_test_cpu PASSED in 40.0s Stats over 20 runs: max = 40.0s, min = 6.5s, avg = 14.8s, dev = 10.2s //tensorflow/python/autograph/tests:loop_control_flow_test PASSED in 21.0s Stats over 20 runs: max = 21.0s, min = 13.5s, avg = 18.1s, dev = 2.5s //tensorflow/python/kernel_tests:metrics_test PASSED in 36.0s Stats over 20 runs: max = 36.0s, min = 8.9s, avg = 18.3s, dev = 7.6s //tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu PASSED in 7.0s Stats over 20 runs: max = 7.0s, min = 3.0s, avg = 4.8s, dev = 1.5s //tensorflow/python/kernel_tests/data_structures:barrier_ops_test PASSED in 14.6s Stats over 20 runs: max = 14.6s, min = 3.9s, avg = 7.3s, dev = 2.4s //tensorflow/python/kernel_tests/linalg:eig_op_test PASSED in 40.4s Stats over 20 runs: max = 40.4s, min = 7.6s, avg = 16.2s, dev = 11.5s //tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu PASSED in 92.1s Stats over 20 runs: max = 92.1s, min = 26.8s, avg = 47.6s, dev = 18.3s //tensorflow/python/kernel_tests/linalg:norm_op_test_cpu PASSED in 8.2s Stats over 20 runs: max = 8.2s, min = 4.5s, avg = 6.6s, dev = 1.3s //tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu PASSED in 13.6s Stats over 20 runs: max = 13.6s, min = 5.5s, avg = 10.0s, dev = 2.0s //tensorflow/python/kernel_tests/linalg:qr_op_test_cpu PASSED in 121.6s Stats over 20 runs: max = 121.6s, min = 28.5s, avg = 78.8s, dev = 31.4s //tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu PASSED in 21.8s Stats over 20 runs: max = 21.8s, min = 7.3s, avg = 11.8s, dev = 4.5s //tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu PASSED in 22.1s Stats over 20 runs: max = 22.1s, min = 5.8s, avg = 14.2s, dev = 5.3s //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu PASSED in 28.7s Stats over 20 runs: max = 28.7s, min = 15.3s, avg = 21.6s, dev = 3.7s //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu PASSED in 60.3s Stats over 20 runs: max = 60.3s, min = 5.1s, avg = 25.5s, dev = 17.9s //tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu PASSED in 22.5s Stats over 20 runs: max = 22.5s, min = 14.3s, avg = 16.9s, dev = 1.8s //tensorflow/python/data/experimental/kernel_tests/service:local_workers_test PASSED in 20.9s Stats over 24 runs: max = 20.9s, min = 8.4s, avg = 14.6s, dev = 3.2s //tensorflow/python/data/kernel_tests:interleave_test PASSED in 17.7s Stats over 24 runs: max = 17.7s, min = 6.3s, avg = 11.4s, dev = 3.6s //tensorflow/python/data/kernel_tests:sample_from_datasets_test PASSED in 23.2s Stats over 24 runs: max = 23.2s, min = 5.4s, avg = 11.9s, dev = 4.8s //tensorflow/compiler/xla/tests:array_elementwise_ops_test_cpu PASSED in 11.8s Stats over 25 runs: max = 11.8s, min = 6.6s, avg = 8.6s, dev = 1.6s //tensorflow/compiler/xla/tests:select_and_scatter_test_cpu PASSED in 36.0s Stats over 25 runs: max = 36.0s, min = 8.8s, avg = 13.7s, dev = 6.7s //tensorflow/compiler/xla/tests:convolution_variants_test_cpu PASSED in 9.4s Stats over 30 runs: max = 9.4s, min = 7.1s, avg = 8.4s, dev = 0.6s //tensorflow/compiler/xla/tests:iota_test_cpu PASSED in 14.6s Stats over 30 runs: max = 14.6s, min = 13.2s, avg = 13.6s, dev = 0.3s //tensorflow/compiler/xla/tests:params_test_cpu PASSED in 7.7s Stats over 30 runs: max = 7.7s, min = 5.9s, avg = 6.7s, dev = 0.4s //tensorflow/compiler/xla/tests:reshape_test_cpu PASSED in 7.8s Stats over 30 runs: max = 7.8s, min = 6.7s, avg = 7.3s, dev = 0.3s //tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu PASSED in 16.6s Stats over 30 runs: max = 16.6s, min = 6.0s, avg = 9.2s, dev = 2.8s //tensorflow/compiler/xla/tests:reduce_test_cpu PASSED in 7.3s Stats over 31 runs: max = 7.3s, min = 6.3s, avg = 6.7s, dev = 0.3s //tensorflow/compiler/xla/tests:scalar_computations_test_cpu PASSED in 8.7s Stats over 32 runs: max = 8.7s, min = 6.1s, avg = 7.3s, dev = 0.6s //tensorflow/python/data/experimental/kernel_tests/service:auto_shard_test PASSED in 23.0s Stats over 32 runs: max = 23.0s, min = 6.1s, avg = 16.2s, dev = 4.1s //tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 29.3s Stats over 32 runs: max = 29.3s, min = 9.2s, avg = 17.3s, dev = 4.5s //tensorflow/compiler/xla/tests:batch_normalization_test_cpu PASSED in 8.6s Stats over 40 runs: max = 8.6s, min = 7.1s, avg = 7.7s, dev = 0.4s //tensorflow/compiler/xla/tests:bfloat16_test_cpu PASSED in 7.7s Stats over 40 runs: max = 7.7s, min = 6.6s, avg = 7.1s, dev = 0.3s //tensorflow/compiler/xla/tests:conv_depthwise_backprop_filter_test_cpu PASSED in 9.8s Stats over 40 runs: max = 9.8s, min = 6.8s, avg = 8.0s, dev = 0.8s //tensorflow/compiler/xla/tests:slice_test_cpu PASSED in 10.9s Stats over 40 runs: max = 10.9s, min = 6.0s, avg = 7.3s, dev = 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 123.6s Stats over 50 runs: max = 123.6s, min = 26.3s, avg = 61.3s, dev = 37.1s //tensorflow/compiler/tests:sort_ops_test_cpu PASSED in 38.5s Stats over 50 runs: max = 38.5s, min = 3.3s, avg = 12.3s, dev = 7.3s //tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test PASSED in 43.3s Stats over 50 runs: max = 43.3s, min = 2.9s, avg = 11.9s, dev = 8.7s //tensorflow/compiler/xla/tests:conv_depthwise_test_cpu PASSED in 11.7s Stats over 50 runs: max = 11.7s, min = 7.0s, avg = 9.0s, dev = 1.1s //tensorflow/compiler/xla/tests:convolution_test_1d_no_vmodule_cpu PASSED in 12.3s Stats over 50 runs: max = 12.3s, min = 8.9s, avg = 10.6s, dev = 0.8s //tensorflow/compiler/xla/tests:convolution_test_cpu PASSED in 14.8s Stats over 50 runs: max = 14.8s, min = 9.4s, avg = 12.0s, dev = 1.5s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 18.1s Stats over 50 runs: max = 18.1s, min = 4.6s, avg = 11.1s, dev = 3.7s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_grad_test_cpu PASSED in 10.2s Stats over 50 runs: max = 10.2s, min = 2.6s, avg = 6.2s, dev = 2.6s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_sparse_mat_mul_grad_test_cpu PASSED in 8.4s Stats over 50 runs: max = 8.4s, min = 3.0s, avg = 4.4s, dev = 1.6s //tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu PASSED in 31.8s Stats over 50 runs: max = 31.8s, min = 7.4s, avg = 15.3s, dev = 6.0s //tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu PASSED in 13.4s Stats over 50 runs: max = 13.4s, min = 3.2s, avg = 5.7s, dev = 2.0s //tensorflow/python/kernel_tests/math_ops:cwise_ops_unary_test_cpu PASSED in 14.7s Stats over 50 runs: max = 14.7s, min = 3.1s, avg = 5.0s, dev = 2.2s Executed 3638 out of 3638 tests: 3634 tests pass and 4 fail locally. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.